VIRTUAL MEMORY

Operating System - Virtual Memory



Virtual memory is a technique that allows the execution of processes which are not completely available in memory. The main visible advantage of this scheme is that programs can be larger than physical memory. Virtual memory is the separation of user logical memory from physical memory.

This separation allows an extremely large virtual memory to be provided for programmers when only a smaller physical memory is available. Following are the situations, when entire program is not required to be loaded fully in main memory.

User written error handling routines are used only when an error occured in the data or computation.

Certain options and features of a program may be used rarely.

Many tables are assigned a fixed amount of address space even though only a small amount of the table is actually used.

The ability to execute a program that is only partially in memory would counter many benefits.

Less number of I/O would be needed to load or swap each user program into memory.

A program would no longer be constrained by the amount of physical memory that is available.

Each user program could take less physical memory, more programs could be run the same time, with a corresponding increase in CPU utilization and throughput.

Virtual Memory


Virtual memory is commonly implemented by demand paging. It can also be implemented in a segmentation system. Demand segmentation can also be used to provide virtual memory.

Demand Paging

A demand paging system is quite similar to a paging system with swapping. When we want to execute a process, we swap it into memory. Rather than swapping the entire process into memory, however, we use a lazy swapper called pager.

When a process is to be swapped in, the pager guesses which pages will be used before the process is swapped out again. Instead of swapping in a whole process, the pager brings only those necessary pages into memory. Thus, it avoids reading into memory pages that will not be used in anyway, decreasing the swap time and the amount of physical memory needed.

Hardware support is required to distinguish between those pages that are in memory and those pages that are on the disk using the valid-invalid bit scheme. Where valid and invalid pages can be checked by checking the bit. Marking a page will have no effect if the process never attempts to access the page. While the process executes and accesses pages that are memory resident, execution proceeds normally.

Demand Paging

Access to a page marked invalid causes a page-fault trap. This trap is the result of the operating system's failure to bring the desired page into memory. But page fault can be handled as following

Page Fault

Steps

Step 1 Check an internal table for this process, to determine whether the reference was a valid or it was an invalid memory access.
Step 2 If the reference was invalid, terminate the process. If it was valid, but page have not yet brought in, page in the latter.
Step 3 Find a free frame.
Step 4 Schedule a disk operation to read the desired page into the newly allocated frame.
Step 5 When the disk read is complete, modify the internal table kept with the process and the page table to indicate that the page is now in memory.
Step 6 Restart the instruction that was interrupted by the illegal address trap. The process can now access the page as though it had always been in memory. Therefore, the operating system reads the desired page into memory and restarts the process as though the page had always been in memory.

ADVANTAGES

Following are the advantages of Demand Paging

Large virtual memory.

More efficient use of memory.

Unconstrained multiprogramming. There is no limit on degree of multiprogramming.

DISADVANTAGES


Following are the disadvantages of Demand Paging

Number of tables and amount of processor overhead for handling page interrupts are greater than in the case of the simple paged management techniques.

Due to the lack of an explicit constraints on a jobs address space size.

Page Replacement Algorithm

Page replacement algorithms are the techniques using which Operating System decides which memory pages to swap out, write to disk when a page of memory needs to be allocated. Paging happens whenever a page fault occurs and a free page cannot be used for allocation purpose accounting to reason that pages are not available or the number of free pages is lower than required pages.

When the page that was selected for replacement and was paged out, is referenced again then it has to read in from disk, and this requires for I/O completion. This process determines the quality of the page replacement algorithm: the lesser the time waiting for page-ins, the better is the algorithm. A page replacement algorithm looks at the limited information about accessing the pages provided by hardware, and tries to select which pages should be replaced to minimize the total number of page misses, while balancing it with the costs of primary storage and processor time of the algorithm itself. There are many different page replacement algorithms. We evaluate an algorithm by running it on a particular string of memory reference and computing the number of page faults.

Reference String


The string of memory references is called reference string. Reference strings are generated artificially or by tracing a given system and recording the address of each memory reference. The latter choice produces a large number of data, where we note two things.

For a given page size we need to consider only the page number, not the entire address.

If we have a reference to a page p, then any immediately following references to page p will never cause a page fault. Page p will be in memory after the first reference; the immediately following references will not fault.

For example, consider the following sequence of addresses - 123,215,600,1234,76,96

If page size is 100 then the reference string is 1,2,6,12,0,0

First In First Out (FIFO) algorithm

Oldest page in main memory is the one which will be selected for replacement.

Easy to implement, keep a list, replace pages from the tail and add new pages at the head.

First In First Out

Optimal Page algorithm
An optimal page-replacement algorithm has the lowest page-fault rate of all algorithms. An optimal page-replacement algorithm exists, and has been called OPT or MIN.

Replace the page that will not be used for the longest period of time . Use the time when a page is to be used.

Optimal page replacement
Least Recently Used (LRU) algorithm
Page which has not been used for the longest time in main memory is the one which will be selected for replacement.

Easy to implement, keep a list, replace pages by looking back into time.

Least Recently Used
Page Buffering algorithm
To get process start quickly, keep a pool of free frames.

On page fault, select a page to be replaced.

Write new page in the frame of free pool, mark the page table and restart the process.

Now write the dirty page out of disk and place the frame holding replaced page in free pool.

Least frequently Used(LFU) algorithm


Page with the smallest count is the one which will be selected for replacement.

This algorithm suffers from the situation in which a page is used heavily during the initial phase of a process, but then is never used again.

Most frequently Used(LFU) algorithm

This algorithm is based on the argument that the page with the smallest count was probably just brought in and has yet to be used.

Memory Management

OS - Memory Management

Memory management is the functionality of an operating system which handles or manages primary memory. Memory management keeps track of each and every memory location either it is allocated to some process or it is free. It checks how much memory is to be allocated to processes. It decides which process will get memory at what time. It tracks whenever some memory gets freed or unallocated and correspondingly it updates the status.

Memory management provides protection by using two registers, a base register and a limit register. The base register holds the smallest legal physical memory address and the limit register specifies the size of the range. For example, if the base register holds 300000 and the limit register is 1209000, then the program can legally access all addresses from 300000 through 411999.

Memory Management
Instructions and data to memory addresses can be done in following ways

Compile time -- When it is known at compile time where the process will reside, compile time binding is used to generate the absolute code.

Load time -- When it is not known at compile time where the process will reside in memory, then the compiler generates re-locatable code.

Execution time -- If the process can be moved during its execution from one memory segment to another, then binding must be delayed to be done at run time

Dynamic Loading
In dynamic loading, a routine of a program is not loaded until it is called by the program. All routines are kept on disk in a re-locatable load format. The main program is loaded into memory and is executed. Other routines methods or modules are loaded on request. Dynamic loading makes better memory space utilization and unused routines are never loaded.

Dynamic Linking
Linking is the process of collecting and combining various modules of code and data into a executable file that can be loaded into memory and executed. Operating system can link system level libraries to a program. When it combines the libraries at load time, the linking is called static linking and when this linking is done at the time of execution, it is called as dynamic linking.

In static linking, libraries linked at compile time, so program code size becomes bigger whereas in dynamic linking libraries linked at execution time so program code size remains smaller.

Logical versus Physical Address Space
An address generated by the CPU is a logical address whereas address actually available on memory unit is a physical address. Logical address is also known a Virtual address.

Virtual and physical addresses are the same in compile-time and load-time address-binding schemes. Virtual and physical addresses differ in execution-time address-binding scheme.

The set of all logical addresses generated by a program is referred to as a logical address space. The set of all physical addresses corresponding to these logical addresses is referred to as a physical address space.

The run-time mapping from virtual to physical address is done by the memory management unit (MMU) which is a hardware device. MMU uses following mechanism to convert virtual address to physical address.

The value in the base register is added to every address generated by a user process which is treated as offset at the time it is sent to memory. For example, if the base register value is 10000, then an attempt by the user to use address location 100 will be dynamically reallocated to location 10100.

The user program deals with virtual addresses; it never sees the real physical addresses.

Swapping
Swapping is a mechanism in which a process can be swapped temporarily out of main memory to a backing store , and then brought back into memory for continued execution.

Backing store is a usually a hard disk drive or any other secondary storage which fast in access and large enough to accommodate copies of all memory images for all users. It must be capable of providing direct access to these memory images.

Major time consuming part of swapping is transfer time. Total transfer time is directly proportional to the amount of memory swapped. Let us assume that the user process is of size 100KB and the backing store is a standard hard disk with transfer rate of 1 MB per second. The actual transfer of the 100K process to or from memory will take

100KB / 1000KB per second

= 1/10 second

= 100 milliseconds

Process Swapping
Memory Allocation

Main memory usually has two partitions

Low Memory -- Operating system resides in this memory.

High Memory -- User processes then held in high memory.

Operating system uses the following memory allocation mechanism.


1 Single-partition allocation In this type of allocation, relocation-register scheme is used to protect user processes from each other, and from changing operating-system code and data. Relocation register contains value of smallest physical address whereas limit register contains range of logical addresses. Each logical address must be less than the limit register.

2 Multiple-partition allocation In this type of allocation, main memory is divided into a number of fixed-sized partitions where each partition should contain only one process. When a partition is free, a process is selected from the input queue and is loaded into the free partition. When the process terminates, the partition becomes available for another process.
Fragmentation
As processes are loaded and removed from memory, the free memory space is broken into little pieces. It happens after sometimes that processes can not be allocated to memory blocks considering their small size and memory blocks remains unused. This problem is known as Fragmentation.

Fragmentation is of two types


1 External fragmentation Total memory space is enough to satisfy a request or to reside a process in it, but it is not contiguous so it can not be used.
2 Internal fragmentation Memory block assigned to process is bigger. Some portion of memory is left unused as it can not be used by another process.
External fragmentation can be reduced by compaction or shuffle memory contents to place all free memory together in one large block. To make compaction feasible, relocation should be dynamic.

Paging
External fragmentation is avoided by using paging technique. Paging is a technique in which physical memory is broken into blocks of the same size called pages (size is power of 2, between 512 bytes and 8192 bytes). When a process is to be executed, it's corresponding pages are loaded into any available memory frames.

Logical address space of a process can be non-contiguous and a process is allocated physical memory whenever the free memory frame is available. Operating system keeps track of all free frames. Operating system needs n free frames to run a program of size n pages.

Address generated by CPU is divided into

Page number (p) -- page number is used as an index into a page table which contains base address of each page in physical memory.

Page offset (d) -- page offset is combined with base address to define the physical memory address.

Paging
Following figure show the paging table architecture.

Paging Example



Segmentation

Segmentation is a technique to break memory into logical pieces where each piece represents a group of related information. For example ,data segments or code segment for each process, data segment for operating system and so on. Segmentation can be implemented using or without using paging.

Unlike paging, segment are having varying sizes and thus eliminates internal fragmentation. External fragmentation still exists but to lesser extent.

Logical Address Space
Address generated by CPU is divided into

Segment number (s) -- segment number is used as an index into a segment table which contains base address of each segment in physical memory and a limit of segment.

Segment offset (o) -- segment offset is first checked against limit and then is combined with base address to define the physical memory address.

Segmentation Example

PROCESS MANAGEMENT

PROCESS MANAGEMENT

Process

A process is a program in execution. The execution of a process must progress in a sequential fashion. Definition of process is following.

A process is defined as an entity which represents the basic unit of work to be implemented in the system.

Components of process are following.


1 Object Program
Code to be executed.


2 Data
Data to be used for executing the program.
3 Resources
While executing the program, it may require some resources.

4 Status
Verifies the status of the process execution.A process can run to completion only when all requested resources have been allocated to the process. Two or more processes could be executing the same program, each using their own data and resources.

Program

A program by itself is not a process. It is a static entity made up of program statement while process is a dynamic entity. Program contains the instructions to be executed by processor.

A program takes a space at single place in main memory and continues to stay there. A program does not perform any action by itself.

Process States
As a process executes, it changes state. The state of a process is defined as the current activity of the process.

Process can have one of the following five states at a time.


1 New
The process is being created.

2 Ready
The process is waiting to be assigned to a processor. Ready processes are waiting to have the processor allocated to them by the operating system so that they can run.

3 Running
Process instructions are being executed (i.e. The process that is currently being executed).

4 Waiting
The process is waiting for some event to occur (such as the completion of an I/O operation).

5 Terminated
The process has finished execution.

Process States

Process Control Block, PCB

Each process is represented in the operating system by a process control block (PCB) also called a task control block. PCB is the data structure used by the operating system. Operating system groups all information that needs about particular process.

PCB contains many pieces of information associated with a specific process which are described below.


1 Pointer
Pointer points to another process control block. Pointer is used for maintaining the scheduling list.

2 Process State
Process state may be new, ready, running, waiting and so on.

3 Program Counter
Program Counter indicates the address of the next instruction to be executed for this process.

4 CPU registers
CPU registers include general purpose register, stack pointers, index registers and accumulators etc. number of register and type of register totally depends upon the computer architecture.

5 Memory management information
This information may include the value of base and limit registers, the page tables, or the segment tables depending on the memory system used by the operating system. This information is useful for deallocating the memory when the process terminates.

6 Accounting information
This information includes the amount of CPU and real time used, time limits, job or process numbers, account numbers etc.

Process Control Block

Process control block includes CPU scheduling, I/O resource management, file management information etc.. The PCB serves as the repository for any information which can vary from process to process. Loader/linker sets flags and registers when a process is created. If that process get suspended, the contents of the registers are saved on a stack and the pointer to the particular stack frame is stored in the PCB. By this technique, the hardware state can be restored so that the process can be scheduled to run again.


Process Scheduling

Definition

The process scheduling is the activity of the process manager that handles the removal of the running process from the CPU and the selection of another process on the basis of a particular strategy.

Process scheduling is an essential part of a Multiprogramming operating system. Such operating systems allow more than one process to be loaded into the executable memory at a time and loaded process shares the CPU using time multiplexing.


Scheduling Queues

Scheduling queues refers to queues of processes or devices. When the process enters into the system, then this process is put into a job queue. This queue consists of all processes in the system. The operating system also maintains other queues such as device queue. Device queue is a queue for which multiple processes are waiting for a particular I/O device. Each device has its own device queue.

This figure shows the queuing diagram of process scheduling.

Queue is represented by rectangular box.

The circles represent the resources that serve the queues.

The arrows indicate the process flow in the system.

Queuing Diagram

Queues are of two types

Ready queue

Device queue

A newly arrived process is put in the ready queue. Processes waits in ready queue for allocating the CPU. Once the CPU is assigned to a process, then that process will execute. While executing the process, any one of the following events can occur.

The process could issue an I/O request and then it would be placed in an I/O queue.

The process could create new sub process and will wait for its termination.

The process could be removed forcibly from the CPU, as a result of interrupt and put back in the ready queue.

Two State Process Model

Two state process model refers to running and non-running states which are described below.


1 Running
When new process is created by Operating System that process enters into the system as in the running state.

2 Not Running
Processes that are not running are kept in queue, waiting for their turn to execute. Each entry in the queue is a pointer to a particular process. Queue is implemented by using linked list. Use of dispatcher is as follows. When a process is interrupted, that process is transferred in the waiting queue. If the process has completed or aborted, the process is discarded. In either case, the dispatcher then selects a process from the queue to execute.

Schedulers

Schedulers are special system softwares which handles process scheduling in various ways.Their main task is to select the jobs to be submitted into the system and to decide which process to run. Schedulers are of three types

Long Term Scheduler

Short Term Scheduler

Medium Term Scheduler

Long Term Scheduler

It is also called job scheduler. Long term scheduler determines which programs are admitted to the system for processing. Job scheduler selects processes from the queue and loads them into memory for execution. Process loads into the memory for CPU scheduling. The primary objective of the job scheduler is to provide a balanced mix of jobs, such as I/O bound and processor bound. It also controls the degree of multiprogramming. If the degree of multiprogramming is stable, then the average rate of process creation must be equal to the average departure rate of processes leaving the system.

On some systems, the long term scheduler may not be available or minimal. Time-sharing operating systems have no long term scheduler. When process changes the state from new to ready, then there is use of long term scheduler.

Short Term Scheduler

It is also called CPU scheduler. Main objective is increasing system performance in accordance with the chosen set of criteria. It is the change of ready state to running state of the process. CPU scheduler selects process among the processes that are ready to execute and allocates CPU to one of them.

Short term scheduler also known as dispatcher, execute most frequently and makes the fine grained decision of which process to execute next. Short term scheduler is faster than long term scheduler.

Medium Term Scheduler

Medium term scheduling is part of the swapping. It removes the processes from the memory. It reduces the degree of multiprogramming. The medium term scheduler is in-charge of handling the swapped out-processes.

Medium Term Scheduler

Running process may become suspended if it makes an I/O request. Suspended processes cannot make any progress towards completion. In this condition, to remove the process from memory and make space for other process, the suspended process is moved to the secondary storage. This process is called swapping, and the process is said to be swapped out or rolled out. Swapping may be necessary to improve the process mix.


Context Switch

A context switch is the mechanism to store and restore the state or context of a CPU in Process Control block so that a process execution can be resumed from the same point at a later time. Using this technique a context switcher enables multiple processes to share a single CPU. Context switching is an essential part of a multitasking operating system features.

When the scheduler switches the CPU from executing one process to execute another, the context switcher saves the content of all processor registers for the process being removed from the CPU, in its process descriptor. The context of a process is represented in the process control block of a process.

Context switch time is pure overhead. Context switching can significantly affect performance as modern computers have a lot of general and status registers to be saved. Content switching times are highly dependent on hardware support. Context switch requires ( n + m ) bxK time units to save the state of the processor with n general registers, assuming b are the store operations are required to save n and m registers of two process control blocks and each store instruction requires K time units.

Context Switch

Some hardware systems employ two or more sets of processor registers to reduce the amount of context switching time. When the process is switched, the following information is stored.

Program Counter

Scheduling Information

Base and limit register value

Currently used register

Changed State

I/O State

Accounting


Scheduling algorithms

'll discuss four major scheduling algorithms here which are following

First Come First Serve (FCFS) Scheduling

Shortest-Job-First (SJF) Scheduling

Priority Scheduling

Round Robin(RR) Scheduling

Multilevel Queue Scheduling

First Come First Serve (FCFS)

Jobs are executed on first come, first serve basis.

Easy to understand and implement.

Poor in performance as average wait time is high.

First Come First Serve Scheduling Algorithm
Wait time of each process is following

Process Wait Time : Service Time - Arrival Time
P0 0 - 0 = 0
P1 5 - 1 = 4
P2 8 - 2 = 6
P3 16 - 3 = 13
Average Wait Time: (0+4+6+13) / 4 = 5.55

Shortest Job First (SJF)

Best approach to minimize waiting time.

Impossible to implement

Processer should know in advance how much time process will take.

Shortest Job First Scheduling Algorithm
Wait time of each process is following

Process Wait Time : Service Time - Arrival Time
P0 3 - 0 = 3
P1 0 - 0 = 0
P2 16 - 2 = 14
P3 8 - 3 = 5
Average Wait Time: (3+0+14+5) / 4 = 5.50

Priority Based Scheduling

Each process is assigned a priority. Process with highest priority is to be executed first and so on.

Processes with same priority are executed on first come first serve basis.

Priority can be decided based on memory requirements, time requirements or any other resource requirement.

Priority Scheduling Algorithm
Wait time of each process is following

Process Wait Time : Service Time - Arrival Time
P0 0 - 0 = 0
P1 3 - 1 = 2
P2 8 - 2 = 6
P3 16 - 3 = 13
Average Wait Time: (0+2+6+13) / 4 = 5.25

Round Robin Scheduling

Each process is provided a fix time to execute called quantum.

Once a process is executed for given time period. Process is preempted and other process executes for given time period.

Context switching is used to save states of preempted processes.

Round Robin Scheduling Algorithm
Wait time of each process is following

Process Wait Time : Service Time - Arrival Time
P0 (0-0) + (12-3) = 9
P1 (3-1) = 2
P2 6-2) + (15-9) = 10
P3 (9-3) + (18-12) = 12
Average Wait Time: (9+2+10+12) / 4 = 8.25

Multi Queue Scheduling

Multiple queues are maintained for processes.

Each queue can have its own scheduling algorithms.

Priorities are assigned to each queue.

Multi Queue Scheduling Algorithm

SYSTEM CALLS

System Calls

When a computer is turned on, the program that gets executed first is called the ``operating system.'' It controls pretty much all activity in the computer. This includes who logs in, how disks are used, how memory is used, how the CPU is used, and how you talk with other computers. The operating system we use is called "Unix".
The way that programs talk to the operating system is via ``system calls.'' A system call looks like a procedure call (see below), but it's different -- it is a request to the operating system to perform some activity.

System calls are expensive. While a procedure call can usually be performed in a few machine instructions, a system call requires the computer to save its state, let the operating system take control of the CPU, have the operating system perform some function, have the operating system save its state, and then have the operating system give control of the CPU back to you. This concept is important, and will be seen time and time again in this class.
System calls are system dependent. Knowing this, it would not be a good idea to directly use system calls when portability cannot be neglected. System calls are also quite complex. Most often, it involves the duo of TRAP and RET (or some variations of those two). To implement system call, one would need specialized knowledge of I/O registers, the sequence of operations needed to use them; and most important of all, implement enough protection because I/O resources are generally shared among multiple users and/or processes.
System Calls for I/O

There are 5 basic system calls that Unix provides for file I/O.
        1.  int open(char *path, int flags [ , int mode ] );
        2.  int close(int fd);
        3.  int read(int fd, char *buf, int size);
        4.  int write(int fd, char *buf, int size);
        5.  off_t lseek(int fd, off_t offset, int whence);
You'll note that they look like regular procedure calls. This is how you program with them -- like regular procedure calls. However, you should know that they are different: A system call makes a request to the operating system. A procedure call just jumps to a procedure defined elsewhere in your program. That procedure call may itself make a system call (for example, fopen() calls open()), but it is a different call.
The reason the operating system controls I/O is for safety -- the computer must ensure that if my program has a bug in it, then it doesn't crash the system, and it doesn't mess up other people's programs that may be running at the same time or later. Thus, whenever you do disk or screen or network I/O, you must go through the operating system and use system calls.

These five system calls are defined fully in their man pages (do 'man -s 2 open', 'man -s 2 close', etc).

Open

Open makes a request to the operating system to use a file. The 'path' argument specifies what file you would like to use, and the 'flags' and 'mode' arguments specify how you would like to use it. If the operating system approves your request, it will return a ``file descriptor'' to you. This is a non-negative integer. If it returns -1, then you have been denied access, and you have to check the value of the variable "errno" to determine why. (That or use perror() -- see the lecture notes for Chapter 1).
All actions that you will perform on files will be done through the operating system. Whenever you want to do file I/O, you specify the file by its file descriptor. Thus, whenever you want to do file I/O on a specific file, you must first open that file to get a file descriptor.

Example: o1.c opens the file in1 for reading, and prints the value of the file descriptor. If you haven't copied over the file in1, then it will print -1, since in1 does not exist. If in1 does exist, then it will print 3, meaning that the open() request has been granted (i.e. a non-negative integer was returned). 3 is the file descriptor you obtained. Why 3? For each process, three I/O streams are opened by default, they are stdin, stdout and stderr. These three takes the file descriptors 0, 1 and 2.

There is a limit on how many files a program can open, because it takes resources to store all information needed to correctly handle an opened file. So, close all files you don't currently need to save some resources. As to the question of how many files can be opened, the answer varies from system to system and user to user. Some system allows as many as 2000 files, and sometimes a user's resource is limited. (Try running limit and find out). On the Sun machines we have, I found the limit to be 256 files, while a newer linux machine tells me that I can open 1024 files at the same time. Note: technically, the set of system constants that are assumed, i.e. the limits, can be found by tracing /usr/include/limits.h. However, great care must be taken since sometimes the necessary documentation is not available.

Note the value of 'flags' -- the man page for open() (or chapter 3 of the book) will give you a description of the flags and how they work. They are described in fcntl.h, which can be found in the directory /usr/include. (Note that fcntl.h merely includes /usr/include/sys/fcntl.h, so you'll have to look at that file to see what O_RDONLY and all really mean).

Example: o2.c tries to open the file "out1" for writing. That fails because out1 does not exist already. In order to open a new file for writing, you should open it with (O_WRONLY | O_CREAT | O_TRUNC) as the flags argument. See o3.c for an example of that. Notice that it creates the file out2, which is of zero length when the program terminates. Note also how o2.c and o3.c use perror() to flag errors.

UNIX> o2
o2: No such file or directory
UNIX> o3
UNIX> ls -l out*
-rw-r--r--  1 plank           0 Sep 11 08:50 out2
UNIX>
Finally, the 'mode' argument should only be used if you are creating a new file. It specifies the protection mode of the new file. 0644 is the most typical value -- it says "I can read and write it; everyone else can only read it"
You can open the same file more than once. You will get a different fd each time. If you open the same file for writing more than once at a time, you may get bizarre results.

Close

Close() tells the operating system that you are done with a file descriptor. The OS can then reuse that file descriptor. The file c1.c shows some examples with opening and closing the file in1. You should look at it carefully, as it opens the file multiple times without closing it, which is perfectly legal in Unix.
Read

Read() tells the operating system to read "size" bytes from the file opened in file descriptor "fd", and to put those bytes into the location pointed to by "buf". It returns how many bytes were actually read. Consider the code in r1.c. When executed, you get the following:
UNIX> cat in1
Jim Plank
Claxton 221
UNIX> r1
called read(3, c, 10).  returned that 10 bytes  were read.
Those bytes are as follows: Jim Plank

called read(3, c, 99).  returned that 12 bytes  were read.
Those bytes are as follows: Claxton 221

UNIX>
There are a few things to note about this program. First, buf should point to valid memory. In r1.c, this is achieved by calloc()-ing space for c (read the man page on calloc() if you have not seen it before. Alternatively, I could have declared c to be a static array with 100 characters:
  char c[100];
Second, I null terminate c after the read() calls to ensure that printf() will understand it.
Third, when read() returns 0, then the end of file has been reached. When you are reading from a file, if read() returns fewer bytes than you requested, then you have reached the end of the file as well. This is what happens in the second read() in r1.c.

Finally, note that the 10th character in the first read() call and the 12th character in the second are both newline characters. That is why you get two newlines in the printf() statement. One is in c, and the other is in the printf() statement.

Write

Write() is just like read(), only it writes the bytes instead of reading them. It returns the number of bytes actually written, which is almost invariably "size".
w1.c writes the string "cs360\n" to the file out3.

UNIX> w1
called write(3, "cs360\n", 6).  it returned 6
UNIX> cat out3
cs360
UNIX>
Lseek

All open files have a "file pointer" associated with them. When the file is opened, the file pointer points to the beginning of the file. As the file is read or written, the file pointer moves. For example, in r1.c, after the first read, the file pointer points to the 11th byte in in1. You can move the file pointer manually with lseek(). The 'whence' variable of lseek specifies how the seek is to be done -- from the beginning of the file, from the current value of the pointer, and from the end of the file. The return value is the offset of the pointer after the lseek. Look at l1.c It does a bunch of seeks on the file in1. Trace it and make sure it all makes sense. How did I know to include sys/types.h and unistd.h? I typed "man -s 2 lseek".
Standard Input, Standard Output, and Standard Error

Now, every process in Unix starts out with three file descriptors predefined:
File descriptor 0 is standard input.
File descriptor 1 is standard output.
File descriptor 2 is standard error.
Thus, when you write a program, you can read from standard input, using read(0, ...), and write to standard output using write(1, ...).
Thus, we can write a very simple cat program (one that copies standard input to standard output) with one line: (this is in simpcat.c):

main()
{
  char c;

  while (read(0, &c, 1) == 1) write(1, &c, 1);
}
UNIX> simpcat < in1
Jim Plank
Claxton 221
UNIX>
We can have several opened file descriptors pointing to the same file. Suppose, we already have a file tfile on disk, that looks like the following:
UNIX> cat tfile
hope not
The following code, opens tfile twice, write once and read once via two different file descriptors.

#include < string.h >
#include < unistd.h >
#include < fcntl.h >

int main (void)
{
    int fd[2];
    char buf1[12] = "just a test";
    char buf2[12];

    fd[0] = open("tfile",O_RDWR);
    fd[1] = open("tfile",O_RDWR);
   
    write(fd[0],buf1,strlen(buf1));
    write(1, buf2, read(fd[1],buf2,12));

    close(fd[0]);
    close(fd[1]);

    return 0;
}
The output is then:

UNIX> a.out
just a testUNIX>
In this case, the content of tfile is consistent, regardless of using fd[0] or fd[1].

Now, take a look at the following code:
#include < stdio.h >
#include < stdlib.h >
#include < unistd.h >

int main (void)
{
    int i, r, w;
    char msg[12];
    char buf[2] = {0, 0};

    for (i = 0; i < 3; i ++) {
      if ((r = read(i,buf,1))<0) {
         sprintf(msg,"read  f%d:%s",i,buf);  
         perror(msg);
      }
      if ((w = write(i,buf,1))<0) {
         sprintf(msg,"write f%d:%s",i,buf);
         perror(msg);
      }
      fprintf(stderr,"%d, r = %d, w = %d, char = %d\n",i,r,w,(int)(buf[0]));
    }

    return 0;
}
This piece of code is interesting because running it differently, we get very different results. In the following, bold letters signify keyboard inputs, while plain characters are the output. The file in1 is a plain text file:

UNIX> cat in1
ABCDEFGHIJK
UNIX> a.out                   |UNIX> a.out < in1               |UNIX> a.out < in1 > out1
A                             |write f0: Bad file descriptor   |write f0: Bad file descriptor
A0, r = 1, w = 1, char = 65   |0, r = 1, w = -1, char = 65     |0, r = 1, w = -1, char = 65
                              |B                               |read  f1: Bad file descriptor
1, r = 1, w = 1, char = 10    |B1, r = 1, w = 1, char = 66     |1, r = -1, w = 1, char = 65
C                             |                                |B
C2, r = 1, w = 1, char = 67   |2, r = 1, w = 1, char = 10      |B2, r = 1, w = 1, char = 66
Who can tell me what is going on?

Finally, let's take a peek at a concept called data driven programming. In the above program, outputting f0, f1, f2, etc. makes our life easy, since there is a consistent pattern. However, wouldn't outputing names like stdin, stdout and stderr be more clear? But then we cannot leverage a neat for loop any more. In fact, in many applications, when the number of possibilities for outputs goes way beyond 3, the code would look like some big mess like below:

  ....

  for (i = 0; i < 3; i++) {

    if (read(...) < 0) {

      switch i {
         case 0: sprintf(msg, "read stdin ...
         case 1:
         case 2:
        ....
      }

      perror....
    }

    if (write(...) < 0) {

      switch i {
         case 0: sprintf(msg, "write stdin ...
         case 1:
         case 2:
        ....
      }

      perror....
    }

  ....
  }

In light of this, take a look at the following instead:

#include < stdio.h >
#include < stdlib.h >
#include < unistd.h >

char msg[6][15] = {"read  stdin",
                   "write stdin",
                   "read  stdout",
                   "write stdout",
                   "read  stderr",
                   "write stderr"};

int main (void)
{
    int i, r, w;
    char buf[2] = {0, 0};

    for (i = 0; i < 3; i ++) {
      if ((r = read(i,buf,1))<0)
         perror(msg[i*2]);
     
      if ((w = write(i,buf,1))<0)
         perror(msg[i*2+1]);

      fprintf(stderr,"%d, r = %d, w = %d, char = %d\n",i,r,w,(int)(buf[0]));
    }

    return 0;
}
The output is then:


UNIX> a.out < in1 > out1
write stdin: Bad file descriptor
0, r = 1, w = -1, char = 65
read  stdout: Bad file descriptor
1, r = -1, w = 1, char = 65
B
B2, r = 1, w = 1, char = 66
UNIX>
We have already talked about distinguishing mechanism and policy. In this program, the output mechanism employed is simply a table lookup. The exact content of each output message should be a "policy" that gets defined separately. The actual code is this program is shortened. With this, we gain simplicity, robustness, and an ability to re-configure (by storing the output messages into a separate file). Hopefully, now you know that CS folks appreciate a very different kind of "simple and stupid", as opposed to how the general public interpret this term. Indeed, elegance in designs is what we strive for.

OPERATING SYSTEM SERVICES

SERVICES  OF  OPERATING SYSTEM

An Operating System provides services to both the users and to the programs.

It provides programs, an environment to execute.

It provides users, services to execute the programs in a convenient manner.

Following are few common services provided by operating systems.

Program execution

I/O operations

File System manipulation

Communication

Error Detection

Resource Allocation

Protection

Program execution
Operating system handles many kinds of activities from user programs to system programs like printer spooler, name servers, file server etc. Each of these activities is encapsulated as a process.

A process includes the complete execution context (code to execute, data to manipulate, registers, OS resources in use). Following are the major activities of an operating system with respect to program management.

Loads a program into memory.

Executes the program.

Handles program's execution.

Provides a mechanism for process synchronization.

Provides a mechanism for process communication.

Provides a mechanism for deadlock handling.

I/O Operation
I/O subsystem comprised of I/O devices and their corresponding driver software. Drivers hides the peculiarities of specific hardware devices from the user as the device driver knows the peculiarities of the specific device.

Operating System manages the communication between user and device drivers. Following are the major activities of an operating system with respect to I/O Operation.

I/O operation means read or write operation with any file or any specific I/O device.

Program may require any I/O device while running.

Operating system provides the access to the required I/O device when required.

File system manipulation
A file represents a collection of related information. Computer can store files on the disk (secondary storage), for long term storage purpose. Few examples of storage media are magnetic tape, magnetic disk and optical disk drives like CD, DVD. Each of these media has its own properties like speed, capacity, data transfer rate and data access methods.

A file system is normally organized into directories for easy navigation and usage. These directories may contain files and other directions. Following are the major activities of an operating system with respect to file management.

Program needs to read a file or write a file.

The operating system gives the permission to the program for operation on file.

Permission varies from read-only, read-write, denied and so on.

Operating System provides an interface to the user to create/delete files.

Operating System provides an interface to the user to create/delete directories.

Operating System provides an interface to create the backup of file system.

Communication
In case of distributed systems which are a collection of processors that do not share memory, peripheral devices, or a clock, operating system manages communications between processes. Multiple processes with one another through communication lines in the network.

OS handles routing and connection strategies, and the problems of contention and security. Following are the major activities of an operating system with respect to communication.

Two processes often require data to be transferred between them.

The both processes can be on the one computer or on different computer but are connected through computer network.

Communication may be implemented by two methods either by Shared Memory or by Message Passing.

Error handling
Error can occur anytime and anywhere. Error may occur in CPU, in I/O devices or in the memory hardware. Following are the major activities of an operating system with respect to error handling.

OS constantly remains aware of possible errors.

OS takes the appropriate action to ensure correct and consistent computing.

Resource Management
In case of multi-user or multi-tasking environment, resources such as main memory, CPU cycles and files storage are to be allocated to each user or job. Following are the major activities of an operating system with respect to resource management.

OS manages all kind of resources using schedulers.

CPU scheduling algorithms are used for better utilization of CPU.

Protection
Considering a computer systems having multiple users the concurrent execution of multiple processes, then the various processes must be protected from each another's activities.

Protection refers to mechanism or a way to control the access of programs, processes, or users to the resources defined by a computer systems. Following are the major activities of an operating system with respect to protection.

OS ensures that all access to system resources is controlled.

OS ensures that external I/O devices are protected from invalid access attempts.

OS provides authentication feature for each user by means of a password

STRUCTURE OF OPERATING SYSTEM

OPERATING SYSTEM STRUCTURE

An operating System (OS) is an intermediary between users and computer hardware. It provides users an environment in which a user can execute programs conveniently and efficiently.

In technical terms, It is a software which manages hardware. An operating System controls the allocation of resources and services such as memory, processors, devices and information.

Definition
An operating system is a program that acts as an interface between the user and the computer hardware and controls the execution of all kinds of programs.

Conceptual view of an Operating System
Following are some of important functions of an operating System.

Memory Management

Processor Management

Device Management

File Management

Security

Control over system performance

Job accounting

Error detecting aids

Coordination between other software and users

Memory Management
Memory management refers to management of Primary Memory or Main Memory. Main memory is a large array of words or bytes where each word or byte has its own address.

Main memory provides a fast storage that can be access directly by the CPU. So for a program to be executed, it must in the main memory. Operating System does the following activities for memory management.

Keeps tracks of primary memory i.e. what part of it are in use by whom, what part are not in use.

In multiprogramming, OS decides which process will get memory when and how much.

Allocates the memory when the process requests it to do so.

De-allocates the memory when the process no longer needs it or has been terminated.

Processor Management
In multiprogramming environment, OS decides which process gets the processor when and how much time. This function is called process scheduling. Operating System does the following activities for processor management.

Keeps tracks of processor and status of process. Program responsible for this task is known as traffic controller.

Allocates the processor(CPU) to a process.

De-allocates processor when processor is no longer required.

Device Management
OS manages device communication via their respective drivers. Operating System does the following activities for device management.

Keeps tracks of all devices. Program responsible for this task is known as the I/O controller.

Decides which process gets the device when and for how much time.

Allocates the device in the efficient way.

De-allocates devices.

File Management
A file system is normally organized into directories for easy navigation and usage. These directories may contain files and other directions. Operating System does the following activities for file management.

Keeps track of information, location, uses, status etc. The collective facilities are often known as file system.

Decides who gets the resources.

Allocates the resources.

De-allocates the resources.

Other Important Activities
Following are some of the important activities that Operating System does.

Security -- By means of password and similar other techniques, preventing unauthorized access to programs and data.

Control over system performance -- Recording delays between request for a service and response from the system.

Job accounting -- Keeping track of time and resources used by various jobs and users.

Error detecting aids -- Production of dumps, traces, error messages and other debugging and error detecting aids.

Coordination between other softwares and users -- Coordination and assignment of compilers, interpreters, assemblers and other software to the various users of the computer systems.

TYPES OF OPERATING SYSTEM

TYPES OF OPERATING SYSTEM

Operating systems are there from the very first computer generation. Operating systems keep evolving over the period of time. Following are few of the important types of operating system which are most commonly used.

Batch operating system
The users of batch operating system do not interact with the computer directly. Each user prepares his job on an off-line device like punch cards and submits it to the computer operator. To speed up processing, jobs with similar needs are batched together and run as a group. Thus, the programmers left their programs with the operator. The operator then sorts programs into batches with similar requirements.

The problems with Batch Systems are following.

Lack of interaction between the user and job.

CPU is often idle, because the speeds of the mechanical I/O devices is slower than CPU.

Difficult to provide the desired priority.

Time-sharing operating systems
Time sharing is a technique which enables many people, located at various terminals, to use a particular computer system at the same time. Time-sharing or multitasking is a logical extension of multiprogramming. Processor's time which is shared among multiple users simultaneously is termed as time-sharing. The main difference between Multiprogrammed Batch Systems and Time-Sharing Systems is that in case of Multiprogrammed batch systems, objective is to maximize processor use, whereas in Time-Sharing Systems objective is to minimize response time.

Multiple jobs are executed by the CPU by switching between them, but the switches occur so frequently. Thus, the user can receives an immediate response. For example, in a transaction processing, processor execute each user program in a short burst or quantum of computation. That is if n users are present, each user can get time quantum. When the user submits the command, the response time is in few seconds at most.

Operating system uses CPU scheduling and multiprogramming to provide each user with a small portion of a time. Computer systems that were designed primarily as batch systems have been modified to time-sharing systems.

Advantages of Timesharing operating systems are following

Provide advantage of quick response.

Avoids duplication of software.

Reduces CPU idle time.

Disadvantages of Timesharing operating systems are following.

Problem of reliability.

Question of security and integrity of user programs and data.

Problem of data communication.

Distributed operating System
Distributed systems use multiple central processors to serve multiple real time application and multiple users. Data processing jobs are distributed among the processors accordingly to which one can perform each job most efficiently.

The processors communicate with one another through various communication lines (such as high-speed buses or telephone lines). These are referred as loosely coupled systems or distributed systems. Processors in a distributed system may vary in size and function. These processors are referred as sites, nodes, computers and so on.

The advantages of distributed systems are following.

With resource sharing facility user at one site may be able to use the resources available at another.

Speedup the exchange of data with one another via electronic mail.

If one site fails in a distributed system, the remaining sites can potentially continue operating.

Better service to the customers.

Reduction of the load on the host computer.

Reduction of delays in data processing.

Network operating System
Network Operating System runs on a server and and provides server the capability to manage data, users, groups, security, applications, and other networking functions. The primary purpose of the network operating system is to allow shared file and printer access among multiple computers in a network, typically a local area network (LAN), a private network or to other networks. Examples of network operating systems are Microsoft Windows Server 2003, Microsoft Windows Server 2008, UNIX, Linux, Mac OS X, Novell NetWare, and BSD.

The advantages of network operating systems are following.

Centralized servers are highly stable.

Security is server managed.

Upgrades to new technologies and hardwares can be easily integrated into the system.

Remote access to servers is possible from different locations and types of systems.

The disadvantages of network operating systems are following.

High cost of buying and running a server.

Dependency on a central location for most operations.

Regular maintenance and updates are required.

Real Time operating System
Real time system is defines as a data processing system in which the time interval required to process and respond to inputs is so small that it controls the environment. Real time processing is always on line whereas on line system need not be real time. The time taken by the system to respond to an input and display of required updated information is termed as response time. So in this method response time is very less as compared to the online processing.

Real-time systems are used when there are rigid time requirements on the operation of a processor or the flow of data and real-time systems can be used as a control device in a dedicated application. Real-time operating system has well-defined, fixed time constraints otherwise system will fail.For example Scientific experiments, medical imaging systems, industrial control systems, weapon systems, robots, and home-applicance controllers, Air traffic control system etc.

There are two types of real-time operating systems.

HARD REAL-TIME SYSTEMS
Hard real-time systems guarantee that critical tasks complete on time. In hard real-time systems secondary storage is limited or missing with data stored in ROM. In these systems virtual memory is almost never found.

SOFT REAL-TIME SYSTEMS
Soft real time systems are less restrictive. Critical real-time task gets priority over other tasks and retains the priority until it completes. Soft real-time systems have limited utility than hard real-time systems.For example, Multimedia, virtual reality, Advanced Scientific Projects like undersea exploration and planetary rovers etc.