Author: Michael Blood
Setting DRBD in Primary / Primary — common commands to sync resync and make changes
Setting DRBD in Primary / Primary — common commands to sync resync and make changes
As we have been setting up our farm with an NFS share the DRBD primary / primary connection between servers is important.
We are setting up a group of /customcommands/ that we will be able to run to help us keep track of all of the common status and maintenance commands we use, but when we have to create, make changes to the structure, sync and resync, recover, grow or move the servers, We need to document our ‘Best Practices’ and how we can recover.
From base Server install
apt-get install gcc make flex
wget http://oss.linbit.com/drbd/8.4/drbd-8.4.1.tar.gz
tar xvfz drbd-8.4.1.tar.gz
cd drbd-8.4.1/
./configure --prefix=/usr --localstatedir=/var --sysconfdir=/etc --with-km
make KDIR=/lib/modules/3.2.0-58-virtual/build
make install
Setup in/etc/drbd.d/disk.res
resource r0 {
protocol C;
syncer { rate 1000M; }
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
become-primary-on both;
}
net {
#requires a clustered filesystem ocfs2 for 2 prmaries, mounted simultaneously
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
cram-hmac-alg sha1;
shared-secret "sharedsanconfigsecret";
}
on server1{
device /dev/drbd0;
disk /dev/xvdb;
address 192.168.100.10:7788;
meta-disk internal;
}
on riofarm-base-san2 {
device /dev/drbd0;
disk /dev/xvdb;
address 192.168.100.11:7788;
meta-disk internal;
}
}
Setup your /etc/hosts
192.168.100.10 server1
192.168.100.11 server2
Setup /etc/hostname with
server1
reboot, verify your settings and SAVE A DRBDVMTEMPLATE clone your VM to a new server called server2
Setup /etc/hostname with
server2
start drbd with /etc/init.d/drbd this will likely try and create the connection, but this is where we are going to ‘play’ to learn the commands and how we can sync, etc.
cat /proc/drbd #shows the status of the connections server1> drbdadm down r0 #turns of the drbdresource and connection server2> drbdadm down r0 #turns of the drbd resource and connection server1> drbdadm -- --force create-md r0 #creates a new set of meta data on the drive, which 'erases drbds memory of the sync status in the past server2> drbdadm -- --force create-md r0 #creates a new set of meta data on the drive, which 'erases drbds memory of the sync status in the past server1> drbdadm up r0 #turns on the drbdresource and connection and they shoudl connect without a problem, with no memory of a past sync history server2> drbdadm up r0 #turns on the drbdresource and connection and they shoudl connect without a problem, with no memory of a past sync history server1> drbdadm -- --clear-bitmap new-current-uuid r0 # this create a new 'disk sync image' essentially telling drbd that the servers are blank so no sync needs to be done both servers are immediately UpToDate/UptoDate in /proc/drbd server1> drbdadm primary r0 server2> drbdadm primary r0 #make both servers primary and now when you put an a filesystem on /dev/drbd0 you will be able to read and write on both systems as though they are local
So, lets do some failure scenarios, Say, we loose a server, it doesn’t matter which one since they are both primaries, in this case though we will say server2 failed. Create a new VM from DRBDVMTEMPLATE which already had drbd made on it with the configuration or create another one using the instructions above.
Open /etc/hostname and set it to
server2
reboot. Make sure /etc/init.d/drbd start is running
server1>watch cat /proc/drbd #watch the status of dtbd, it is very useful and telling about what is happening, you will want DRBD to be Connected Primary/Unknown UpToDate/DUnknown server2>drbdadm down server2>dbadm wipe-md r0 #this is an optional step that is used to wipe out the meta data, I have not seen that it does anything different than creating the metadata using the command below, but it is useful to know the command in case you want to get rid of md on your disk server2>drbdadm -- --force create-md r0 ##this makes sure that their is no partial resync data left over from where you cloned it from server2>drbdadm up r0 # this brings drbd server2 back into the resource and connects them, it will immediately sart syncing you should see SyncSource Primary/Secondary UpToDate/Inconsistent on server1, for me it was soing to to 22 hours for my test of a 1TM (10 MB / second)
Lets get funky, what happens if you stop everything in the middle of a sync
server1>drbdadm down r0 #we shut down the drdb resource that has the most up to date information, on server2 /proc/drbd shows Secondary/Unknown Inconsitent/DUnknown , server2 does not know about server1 any more, but server2 still knows that server2 is inconsitent, (insertable step here could be on server2: drbdadm down ro; drbdadm up ro, with no change to the effect) server1>drbdadm up ro # this brings server1 back on line and /proc/drbd on server1 shows SyncSource, server2 shows SyncTarget, server1 came backup as the UpToDate server, server2 was Inconsistent, it figured it out
Where things started to go wrong and become less ‘syncable’ was when servers were both down and had to be brought back up again separately with a new uuid was created on them separately. so lets simulate that the drbd config fell apart, and we have to put it together again.
server2>drbdadm disconnect ro; drdbadm -- --force create-md r0 ; drbd connect ro; #start the sync process over
Xen Center – import from exported XVA file for restoring – does not create a new VM
Xen Center – import from exported XVA file for restoring – does not create a new VM
Building a backup procedure for Xen Center
This started when defining the procedure for how to backup VMs ‘off site’ in a way that would later allow us to restore them, should some sort of unrecoverable error occur.
The concept is, we want to be able to take a VM, which is currently running and get a backup of some kind which can then be restored at a later point.
First I will explain what I have found to be the way that Xen Center currenlty allows, I dont know what the ‘Best Practice’ is suggested for this procedure, I couldnt find it from searching and so I explored the options available within Xen Center and at the command line
Exporting (stopping a VM first)
This method requires you to stop a VM first, which means that it does not work well for VMs which are already in production, and the method is not a viable “backup” solution for us, but I explain it here to make the distinction between the different types of exporting. This method is something that would work well for backing up defined templates to an offsite location, but would not work well for saving a full running VM in a way that could be restored from an offsite location.
Xen Center allows you to export a VM, which you later import: First, shut down your VM, right click on it and go to Export and follow the steps in the wizard to put the export on your local drive. you can conveniently do an export of multiple stopped VMs. the progress is displayed in the Logs tab for the pool. I told the export process to ‘verify’ the exported files, which added A LONG time to the process, be prepared for this
Once the export is complete, you can move these files where ever you want, to restore, you simply right click on your pool and go to Import select the file from disk, follow the wizard and the VMS will be started (I am not sure what happens with any kind of MAC address collisions here if the same VM you exported is currently running)
Exporting a live VM
It seems reasonable that a currently running VM could not be exported directly to a file, because the VM is running and the changes that occur in a running VM would be inconsitant during the process of the export. here is how we work around this.
In short, here are the steps
- create a snapshot of a vm
- export the snapshot to a file offsite using Xen Center (we are backed up)
- start the restore by creating a vm martyr by installing a new vm or template (hopefully a small )
- destroy the martyrs existing vdi
- import the snapshot from a file (could be on the same server or pool, or a completely new build)
- attach the imported vdi as a bootable disk
- rename from martry to your correct name and start the new VM
WIth more detail:
First take snapshot of the running VM, then go to the Snapshots tab for th VM and export that Snapshot. Follow the wizard and save it to a file.
When it comes time to re import, we have a little preparation we should do and keep track of these numbers
#Find the UUID for the SR that you will be importing the file back to
xe sr-list host=HOSTNAME
>079eb2c5-091d-5c2f-7e84-17705a8158cf
#get a list of all of the CURRENT uuids for the virtual disks on the sr
xe vdi-list sr-uuid=079eb2c5-091d-5c2f-7e84-17705a8158cf|grep uuid
>uuid ( RO) : bb8015b0-0672-45af-aed5-c5308f60b914
>uuid ( RO) : f0b67634-25bc-486d-b38e-0e8294de7df6
>uuid ( RO) : cdc13e40-9ffe-497c-91ff-d426a52aaf2a
Now we import the file. Right click on the host you would like to restore it to and click ‘Import’ THe import process asks you a couple of pieces of information about the restore, host name, network, etc. go through the steps an click finish. The vm will be imported again, the progress will be shows in the Logs tab of the host and pool, when complete, we now have a virtual disk unattached to a VM, which we need to attach to a VM,
Here things are a bit more complex. First we create a VM ‘martyr’, this is what I call a VM, that we create through some other method, soley for the purpose of attaching our reimported snapshot to it. we will take the guts out of whatever VM we create and put the guts from our import into it. on the technical side, we take a VM, disconnect the existing bootable vdi and reconnect the vdi we just imported. I create the VM using a template or install I dont cover that here, but I name it martyr_for_import
#get a list of the latest uuids for the virtual disks on the sr
xe vdi-list sr-uuid=079eb2c5-091d-5c2f-7e84-17705a8158cf|grep uuid
>uuid ( RO) : bb8015b0-0672-45af-aed5-c5308f60b914
>uuid ( RO) : f0b67634-25bc-486d-b38e-0e8294de7df6
>uuid ( RO) : cdc13e40-9ffe-497c-91ff-d426a52aaf2a
>uuid ( RO) : 04a7f80e-e108-4468-9bd3-fada613e9a42
#each time I have done this, the imported uuid is listed last, but I run the list, before and after, just to make sure, in this case my vdi is: 04a7f80e-e108-4468-9bd3-fada613e9a42
#find the current vbds attached to this vm
xe vbd-list vm-label-name-label=martyr_for_import
>uuid ( RO) : b0f4cb5e-5285-bbec-13a3-f581c6e6d287 vm-uuid ( RO): 708b633a-683d-859f-1f1f-bf8495d17fe8 vm-name-label ( RO): martyr_for_import vdi-uuid ( RO): a36d6025-039b-4f6e-9d19-f7eb7d1d4c46 empty ( RO): false device ( RO): xvdd
uuid ( RO) : eb12fdac-c36c-78fa-8eb6-67fa3a5a1d85 vm-uuid ( RO): 708b633a-683d-859f-1f1f-bf8495d17fe8 vm-name-label ( RO): martyr_for_import vdi-uuid ( RO): cdc13e40-9ffe-497c-91ff-d426a52aaf2a empty ( RO): false device ( RO): xvda #shut down the vm xe vm-shutdown uuid=708b633a-683d-859f-1f1f-bf8495d17fe8 #destroy the vdi virtual disk that is attached to our marty as the current xvda vbd xe vdi-destroy uuid=cdc13e40-9ffe-497c-91ff-d426a52aaf2a #you can verify that it has been destroyed and detached by running xe vbd-list vm-label-name-label=martyr_for_import again #now attach our snapshot vdi as a new vbd bottable device as xvda again. (note the bootable=true and type=Disk) xe vbd-create vm-uuid=708b633a-683d-859f-1f1f-bf8495d17fe8 vdi-uuid=04a7f80e-e108-4468-9bd3-fada613e9a42 bootable=true device=xvda type=Disk #okay we are attached (you can verify by running xe vbd-list vm-label-name-label=martyr_for_import again #go ahead and start the vm through your Xen scenter or run this command xe vm-start uuid=708b633a-683d-859f-1f1f-bf8495d17fe8
awk Command to remove Non IP entries from /etc/hosts and /etc/hosts.deny
awk Command to remove Non IP entries from /etc/hosts and /etc/hosts.deny
We had a script automatically adding malicious IPS to our /etc/hosts.deny file on one of our servers.
The script went awry and ended up putting hundreds of thousands of non ip addresses into the file. There were malicious IP addresses mixed in
I used this awk script to clean it up , and remove all of the non ip addresses, and make the list unique.
awk '/ALL/ && $NF ~ /[0-9.]/' /etc/hosts.deny| sort -n -k2 |uniq > /etc/hosts.deny2
once I inspected the /etc/hosts.deny2 I replaced the original
mv /etc/hosts.deny2 /etc/hosts.deny
Mdadm – Failed disk recovery (unreadable disk)
Mdadm – Failed disk recovery (unreadable disk)
Well,
After 9 more months I ran into a nother disk failure. (First disk failure found here https://www.matraex.com/mdadm-failed-disk-recovery/)
But this time, The system was unable to read the disk at all
#fdisk /dev/sdb
This process just hung for a few minutes. It seems I couldn’t simply run a few commands like before to remove and add the disk back to the software RAID. So I had to replace the disk. Before I went to the datacenter I ran
#mdadm /dev/md0 --remove /dev/sdb1
I physically went to our data center, found the disk that showed the failure (it was disk sdb so I ‘assumed’ it was the center disk out of three, but I was able to verify since it was not blinking from normal disk activity. I removed the disk, swapped it out for one that I had sitting their waiting for this to happen, and replaced it. Then I ran a command to make sure the disk was correctly partitioned to be able to fit into the array
#fdisk /dev/sdb
This command did not hang, but responded with cannot read disk. Darn, looks like some error happened within the OS or on the backplane that made it so a newly added disk wasn’t readable. I scheduled a restart on the server later when the server came back up, fdisk could read the disk. It looks like I had used the disk for something before, but since I had put it in my spare disk pile, I knew I could delete it and I partitioned it with one partion to match what the md was expecting (same as the old disk)
#fdisk /dev/sdb
>d 2 -deletes the old partition 2
>d 1 -deletes the old partition 1
>c -creates a new partion
>p – sets the new partion as primary
>1 – sets the new partion as number 1
>> <ENTER> – just press enter to accept the defaults starting cylinder
>> <ENTER> – just press enter to accept the defaults ending cylinder
>> w – write the partion changes to disk
>> Ctrl +c – break out of fdisk
Now the partition is ready to add back to the raid array
#mdadm /dev/md0 –add /dev/sdb1
And we can immediately see the progress
#mdadm /dev/md0 --detail /dev/md0: Version : 00.90.03 Creation Time : Wed Jul 18 00:57:18 2007 Raid Level : raid5 Array Size : 140632704 (134.12 GiB 144.01 GB) Device Size : 70316352 (67.06 GiB 72.00 GB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Feb 22 10:32:01 2014 State : active, degraded, recovering Active Devices : 2 Working Devices : 3 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K Rebuild Status : 0% complete UUID : fe510f45:66fd464d:3035a68b:f79f8e5b Events : 0.537869 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 3 8 17 1 spare rebuilding /dev/sdb1 2 8 33 2 active sync /dev/sdc1
And then to see the progress of rebuilding
#cat /proc/mdadm Personalities : [raid1] [raid6] [raid5] [raid4] md0 : active raid5 sdb1[3] sda1[0] sdc1[2] 140632704 blocks level 5, 64k chunk, algorithm 2 [3/2] [U_U] [==============>......] recovery = 71.1% (50047872/70316352) finish=11.0min speed=30549K/sec md1 : active raid1 sda2[0] 1365440 blocks [2/1] [U_]
Wow in the time I have been blogging this, already 71 percent rebuilt!, but wait! what is this, md1 is failed? I check my monitor and what do I find but another message that shows that md1 failed with the reboot. I was so used to getting the notice saying md0 was down I did not notice that md1 did not come backup with the reboot! How can this be?
It turnd out that sdb was in use on both md1 and md0, but even through sdb could not be read at all on /dev/sdb and /dev/sdb1 failed out of the md0 array, somehow the raid subsystem had not noticed and degraded the md1 array even though the entire sdb disk was not respoding (perhaps sdb2 WAS responding back then just not sdb), who knows at this point. Maybe the errors on the old disk could have been corrected by the reboot if I had tried that before replacing the disk, but that doesn’t matter any more, All I know is that I have to repartion the sdb device in order to support both the md0 and md1 arrays.
I had to wait until sdb finished rebuilding, then remove it from md0, use fdisk to destroy the partitions, build new partitions matching sda and add the disk back to md0 and md1
Deleting Orphaned Disks in Citrix XenServer
Deleting Orphaned Disks in Citrix XenServer
I found that while building my virtual environment with templates ready to deploy I created quite a few templates and snapshots.
I did a pretty good job of deleting the extras when I didn’t need them any more, but in some cases when deleting a VM I no longer needed, I forgot to check the box to delete the snapshots that went WITH that VM.
I could see under the dom0 host -> Storage tab that space was still allocated to the snapshots, (Usage was higher than the combined visible suage of servers and templates, and Virtual allocation was way higher than it should be)
But without a place that listed the snapshots that were taking up space. When looking into the way to delete these orphaned snapshots (and the disk snapshots that went with them) I found some cumbersome command line methods.
Like this old method that someone used - http://blog.appsense.com/2009/11/deleting-orphaned-disks-in-citrix-xenserver-5-5/
After a big more digging, i found that by just clicking on the Local Storage under the domU then clicking on the ‘Storage’ tab under there, I would see a list of all of the storage elements that are allocated. I would see some that were for snapshots without a name. Turns out those were the ones that were orphaned, If they were allocated to a live server the delete button would not be highlighted so I just deleted those old ones.
Resizing a VDI on XenServer using XenCenter and Commandline
Resizing a VDI on XenServer using XenCenter and Commandline
Occassionally I have a need to change the size of a disk, perhaps to allocate more data to the os.
To do this, on the host I unmount the disk
umount /data
Click on the domU server in XenCenter and click on the Storage tab, select the storage item I want to resize and click ‘Detach’
at the command line on one of the dom0 hosts
xe sr-list host=dom0hostname
write down the uuid of the SR which the Virtual Disk was in. (we will use XXXXX-XXXXX-XXXX)
xe vdi-list sr-uuid=XXXXX-XXXXX-XXXX
write down the uuid of the disk that you wanted to resize(we will use YYYY-YYYY-YYYYY)
Also, note that the the virtual-size parameter that shows. VDIs can not be shrunk so you will need a disk size LARGER than the size displayed here.
xe vdi-resize sr-uuid=YYYY-YYYY-YYYYY disk-size=9887654
Two step recipe – upgrading from Postgres 8.4 to 9.3 and then implementing 9.3 hot_standby replication
Two step recipe – upgrading from Postgres 8.4 to 9.3 and then implementing 9.3 hot_standby replication
Upgrade and existing postgresql database from Postgres 8.4 to 9.3 and then implementing a 9.3 hot_standby replication server so all backups and slow select queries can run from it.
The setup: two servers, the current primary database server (will continue to be the primary database server when using 9.3, but we will call it the master in the replication)
First get and install postgres 9.3 using the postgres apt repositories
master and standby> vi /etc/apt/sources.list - deb http://apt.postgresql.org/pub/repos/apt/ UNAME-pgdg main #UNAME EXAMPLES: precise, squeeze, etch, etc master and standby> apt-get update master> apt-get install postgresql-9.3 postgresql-client-9.3 nfs-kernel-server nfs-client standby> apt-get install postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3 nfs-kernel-server nfs-client
Next create some shared directorys via nfs for file based archiving
standby> mkdir -p /data/dbsync/primaryarchive master> mkdir -p /data/dbsync-archiveto master> vi /etc/exports - /var/lib/postgresql/9.3/main 192.168.*.*(ro,sync,no_subtree_check,no_root_squash) standby> vi /etc/exports - /data/dbsync/primaryarchive 192.168.*.*(rw,sync,no_subtree_che master> vi /etc/fstab - SECONDARYSERVERIP:/data/dbsync/primaryarchive /data/dbsync-archiveto nfs ro 0 1 standby>mkdir -p /mnt/livedb standby> mount PRIMARYSERVERIP:/var/lib/postgresql/8.4/main/ /mnt/livedb master> mount /data/dbsync-archiveto
Now, configure postgres on the master to allow replication and restart, put it on port 5433 so there are no conflictw with 8.4
master> vi /etc/postgresql/9.3/main/pg_hba.conf - host replication all SECONDARYSERVERIP trust master> vi /etc/postgresql/9.3/main/postgresql.conf - wal_level=hot_standby - archive_mode = on - port = 5433 - archive_command = 'test -f /data/dbsync-archiveto/archiveable && cp %p /data/dbsync-archiveto/%f' master> /etc/init.d/postgresql restart
Configure postgres on the standby to allow it to run as a hot_standby
standby> vi /etc/postgresql/9.3/main/postgresql.conf -restore_command = a018/usr/lib/postgresql/9.3/bin/pg_standby -d -t /tmp/pgsql.trigger.5432 /data/dbsync/primaryarchive %f %p %r 2>>/var/log/postgresql/standby.log -recovery_end_command = a018rm -f /tmp/pgsql.trigger.5432 - wal_level=hot_standby - hot_standby = on standby> /etc/init.d/postgresql stop
Now lets get a base backup on the standby
standby> mv /var/lib/postgresql/9.3/main /var/lib/postgresql/9.3/main.old standby>cd /var/lib/postgres/9.3; mv main main.old; standby> pg_basebackup -D main -R -h192.168.120.201 -p5433 -x -Upostgres standby> chown postgres.postgres main/ -R standby> /etc/init.d/postgres start
Thats it!!, you should not have a working replication server
primary> create table tmp as select now(); secondary> select * from tmp;
#check the progress several ways. postregres log, which files and recovery are running and by being able to connect and see updates from the master, on the secondary
standby> tail /var/log/postgresql/postgresql-9.3-main.log
standby> grep 'database system is ready to accept read only connections'
standby> ps ax|grep post
- postgres: wal receiver process streaming 3/43000000
master> psql -Upostgres -c 'select pg_switch_xlog()'
and the log file would switch in the recovery file
standby> ps ax|grep post
- postgres: startup process recovering 000000010000000300000037
That was all to make sure that the replication is working on 9.3, now that I am comfortable with it working, I am going to turn off the replication, copy the data from 8.4 to 9.3 and recreate the replication
First lets stop the postgresql daemon on the standby server so the VERY heavy load from copying the db from 8.4 to 9.3 is not duplication
standby> /etc/init.d/postgresql stop
Next, copy the database from 8.4 to 9.3, I have heard there may be some problems for conversion of some objects between 8.4 and 9.3 but not for me, this went great.
master> pg_dump -C -Upostgres mydatabase| psql -Upostgres -p5433
Once that is successful, lets switch ports on the 9.3 and 8.4 servers so 9.3 can take over
master>vi /etc/postgresql/9.3/main/postgresql.conf - port = 5432 master>vi /etc/postgresql/8.4/main/postgresql.conf - port = 5433 master> /etc/init.d/postgres reload Last step, get a base backup and start again. standby> mv /var/lib/postgresql/9.3/main /var/lib/postgresql/9.3/main.old standby>cd /var/lib/postgres/9.3; mv main main.old; standby> pg_basebackup -D main -R -h192.168.120.201 -x -Upostgres standby> chown postgres.postgres main/ -R standby> /etc/init.d/postgres start standby> rm /var/lib/postgres/9.3/main.old* -rf Now..... to figure out what to do with the archivedir method we are currently using..... It seems that it is just building up when do we use it?
Employee Appreciation Night – Wahooz
Employee Appreciation Night – Wahooz
The tema at Matraex went to Wahooz on Tuesday February 4th for some pizza, laser tag, go carts and video games.
We ended up with a group of 15 with family and all.
I was lots of fun and we will probably end up doing it again!
PHP to reset all primary key sequences in your postgresql database
PHP to reset all primary key sequences in your postgresql database
Use the following php code t reset all of the primary key sequences with the max(id) currently in the db.
We use wrapper functions db_query (which returns an array from the db when a select statement is run) and db_exec() which runs an update or insert command against the db.
[code language=”php”]$sql = "SELECT t.relname as related_table,
a.attname as related_column,
s.relname as sequence_name
FROM pg_class s
JOIN pg_depend d ON d.objid = s.oid
JOIN pg_class t ON d.objid = s.oid AND d.refobjid = t.oid
JOIN pg_attribute a ON (d.refobjid, d.refobjsubid) = (a.attrelid, a.attnum)
JOIN pg_namespace n ON n.oid = s.relnamespace
WHERE s.relkind = ‘S’
AND n.nspname = ‘public’";
$qry = db_query($sql);
foreach($qry as $row)
{
$outsql = "select setval(‘$row[sequence_name]’,(select max($row[related_column]) from $row[related_table]))";
db_exec($outsql);
}[/code]
Setting up postgres warm standby
Setting up postgres warm standby
Mainly just notes, I haven’t gone through this in too much detail yet…. -MB
Setup two postgres servers with the same version, both with the same data directory layout.
AS ROOT
primary> apt-get install postgresql-server nfs-kernel-server nfs-client
secondary> apt-get install postgresql-server nfs-kernel-server nfs-client postgresql-contrib-8.4
secondary> mkdir -p /data/dbsync/primaryarchive
primary> mkdir -p /data/dbsync-archiveto
primary> vi /etc/exports
#to temporarily allow access to the base data for getting a base
/var/lib/postgresql/8.4/main 192.168.*.*(ro,sync,no_subtree_check,no_root_squash)
secondary> vi /etc/exports
#to provide a location on the secondary, that the primary can write WAL logs to
/data/dbsync/primaryarchive 192.168.*.*(rw,sync,no_subtree_che
primary> vi /etc/fstab
SECONDARYSERVERIP:/data/dbsync/primaryarchive /data/dbsync-archiveto nfs ro 0 1
primary> mount /data/dbsync-archiveto
primary> vi /etc/postgresql/8.4/main/postgresql.conf
#to turn on archiving from primary to the secondary
-archive_mode = on
-archive_command ‘cp %p /data/dbsync-archiveto/%f’
secondary>mkdir -p /mnt/livedb
secondary> mount PRIMARYSERVERIP:/var/lib/postgresql/8.4/main/ /mnt/livedb
At this point we should have a postgres db running on both primary and secondary in /var/lib/postgresql/8.4/main
we should have a mount on the primary pointing to the secondary and WAL logs being writted to the secondary. If the mount to the secondary fails the WAL Logs would build up on the primary in /var/lib/postgresql/8.4/main/pg_xlog.
we should have a mount on the secondary pointing to the base install of the primary database so we can copy the base
secondary> /etc/init.d/postgresql stop
secondary> rsync -Cqtar –delete /mnt/livedb/* /var/lib/postgresql/8.4/main/.
secondary> vi /var/lib/postgresql/8.4/main/recovery.conf
While that is going, setup the configuration on secondary to be monitoring the archive directory on primary and processing the archive entries.
secondary> vi /etc/postgresql/8.4/main/postgresql.conf
restore_command = ‘/usr/lib/postgresql/8.4/bin/pg_standby -d -t /tmp/pgsql.trigger.5432 /data/dbsync/primaryarchive %f %p %r 2>>/var/log/postgresql/standby.log’
recovery_end_command = ‘rm -f /tmp/pgsql.trigger.5432’
———————
(the above is ideal, but sometimes NFS mount issues cause problems, here are some shortcuts I have used….)
– use the mount from primary to secondary to do the initial base transfer.
- On the secondary stop postgres (/etc/init.d/postgresql stop)
- mv /var/lib/postgresql/8.4/main to data/dbsync/primaryarchive ( /data/dbsync/primaryarchive/main)
- from the primary start a backup and sync to the secondary
- primary> psql -Upostgres -c “select pg_start_backup(‘backmeup’)” && time rsync –delete -Ctar /var/lib/postgresql/8.4/main/* /data/dbsync-archiveto/main/. –progress –exclude=’*pg_xlog/*’ && psql -Upostgres -c “select pg_stop_backup()”
- mv main back to the running location (mv /data/dbsync/primary/main /var/lib/postgresql/8.4/.)
- make sure all files are property owned by postgres.postgres (chown postgres.postgres -R /var/lib/postgresql/8.4/main)
- vi /var/lib/postgresql/8.4/main/recovery.conf
- restore_command = ‘/usr/lib/postgresql/8.4/bin/pg_standby -d -t /tmp/pgsql.trigger.5432 /data/dbsync/primaryarchive %f %p %r 2>>/var/log/postgresql/standby.log’
recovery_end_command = ‘rm -f /tmp/pgsql.trigger.5432’ - /etc/init.d/postgresql start