Today after some backups, generated from my C program generated few files exactly 2Gb in size I’ve remembered that there was some parameter to open for working with large files. I’ve opened man page for open and read that the option is O_LARGEFILE. I also liked the idea of using O_DIRECT. But when I tried to compile my program I got two errors:

hostbackupd.c: In function `somefunction':
somefile.c:102: error: `O_LARGEFILE' undeclared (first use in this function)
somefile.c:102: error: (Each undeclared identifier is reported only once
somefile.c:102: error: for each function it appears in.)

and

hostbackupd.c: In function `somefunction':
somefile.c:102: error: `O_DIRECT' undeclared (first use in this function)
somefile.c:102: error: (Each undeclared identifier is reported only once
somefile.c:102: error: for each function it appears in.)

After some googling I didn’t find something useful and looked at the .h files in /usr/include. I’ve found that header files are cheching if _GNU_SOURCE is defined and if it is, defines O_DIRECT and O_LARGEFILE. So to use this options you shoud compile your program with

gcc -D_GNU_SOURCE .....

or you shoud put

#define _GNU_SOURCE

before all your includes. Have in mind that this may make your program less portable.

And now some words about O_DIRECT. If you plan to use O_DIRECT your buffers should be aligned to pagesize (or sector size, depending on Linux version) and you should read/write in multiples of pagesize. To create buffer aligned at pagesize with size BUFFER_SIZE you can use code like this:


pagesize=getpagesize();
realbuff=malloc(BUFFER_SIZE+pagesize);
alignedbuff=((((int unsigned)realbuff+pagesize-1)/pagesize)*pagesize);

where pagesize is int and realbuff and alignedbuff are pointers. Realbuff is the pointer you should free() when you finished and alignedbuff is the buffer you shoud use with read/write.

You should also write only by multiples of pagesize. So if you want to create file that is not multiple of pagesize you should ftruncate the file when you finished writing.