View source for Talk:Backup and Restore

=Introduction=
To ensure this MediaWiki's content will not be lost, we created a set of script and put it in $wgResourceBase's extensions/BackupAndRestore directory.

The [[backup.sh]] script, convert all SQL database content into an XML file called: [[XLPLATEST.xml]], and put all uploaded files into the [[images/UploadedFiles/]] directory under the specified $wgResourceBase/ directory.

The [[restore.sh]] script, extract the content of [[XLPLATEST.xml]], and reload all files of the [[images/UploadedFiles/]] directory into the currently running MediaWiki.

Alternatively, the data set could be exported to a SQL file. Since that is running in a separate Docker service, the process will be discussed in a different page.

Data backup practice should follow something like 3-2-1 backup principle:
* 3 Copies of Data
* 2 Types of Storage
* 1 Offsite storage location
There are other practices, which can be found here<ref>Mark Cambell, Why 3-2-1 Backup Sucks, https://www.unitrends.com/blog/3-2-1-backup-sucks, last accessed: August 1st 2021</ref>.

==Importance==
To enable proper data security, convenient backup and restore procedure would be the first layer.

This MediaWiki instance already stores all data under [[mountPoint/images]] and [[mountPoint/mariadb]], these two directories store
all the data relevant to MediaWiki, including its uploaded media files, and textual data stored in the [[MariaDB]] database.

==Implementation==
The default [[up.sh]] shell script is a way to allow users to kickoff the program. This process does the following tasks:

# Before shutting down existing instance of docker services, it first asks the existing instance to backup the data.
# Then, use [[docker-compose down --volumes]] to shut down all the docker services.
# If the [[mountPoint/]] directory does not exist, yet, decompress [[InitialDataPackage.tar.gz]] to the mountPoint/ directory.
# When the mountPoint directory has the initial data content, start docker services, using docker-compose up -d
# The docker-compose program reads [[docker-compose.yml]] file to configure docker services.
# When the docker services start running, take the existing data in the directory, if [[images/UploadedFiles/]] directory or [[XLPLATEST.xml]] are available, these data elements will be imported to the database.

==Instructions==
There are two main parts of Backing up MediaWiki Data, the MySQL(MariabDB) database, and uploaded Media Files.
 '''We strongly advise administrators to first restore the image files, then, load the Database content.'''
 This way, all the images would be referenced correctly. This is based on operational experience.

===Media File Backup===
{{:Backup MediaWiki MediaFiles}}

For more details, see [[MediaWiki's Installation, Backup, Restore, and Recovery]].
===Database Backup===
{{:Backup MediaWiki Database}}

==Periodic Backup==
It is necessary to regularly and automatically create backup data copies. The ideal way of doing such a task is to use [[cron]] job.

The following instructions are designed to configure the [[crontab]] for the [[Ubuntu]] container that runs MediaWiki.

A documented solution can be found here [https://www.mediawiki.org/wiki/User:Flominator/Backup_MW Backup_MW by Flominator].
  
[[Category:Process]]

=Restore=

The process of restoring data after backing up can be found here:

==Restoring XML Data==
To manage large XML data set, see this [[Video/Processing Large XML Wikipedia Dumps that won't fit in RAM in Python without Spark|video]]<ref>{{:Video/Processing Large XML Wikipedia Dumps that won't fit in RAM in Python without Spark}}</ref>. For programmable tools to process XML data, see here<ref>{{:Tool/Data Extraction Tools}}</ref>.
The following instruction should be launched in the host (through docker exec or kubectl exec -it command) of the container that hosts the mediawiki service. Where <code>wikifolder</code> shown in the following instruction should be replaced by the location of where your MediaWiki is installed. They are usually located in <code>/var/www/html/</code>.

 php wikifolder/maintenance/importDump.php --dbpass wikidb_userpassword --quiet --wiki wikidb path-to-dumpfile/dumpfile.xml
 php wikifolder/maintenance/rebuildrecentchanges.php

In our case, if you are importing data in a terminal whose working directory contains the xml file to be loaded, say <code>dumpedData.xml</code>. You may just type:
 php wikifolder/maintenance/importDump.php ./dumpedData.xml

==Restoring SQL Data==
The following instruction should be launched in the host (through docker exec or kubectl exec -it command) of the container that hosts the mariadb/mysql service.

 mysql -u $DATABASE_USER -p $DATABASE_NAME < BACKUP_DATA.sql

===Caveat===
In PKC, we suggest that images or any other independent files to be stored in a separate file service. However, for MediaWiki related operations, the following instructions will work. The reason is that we want PKC to be used for managing hyperlinks, and not other kind of data.

For accessing mysql using command line, you might face various problems. One way to allow access is to override the <code>entrypoint</code> in <code>docker-compose.yml</code> for the MariaDB Docker container by adding the following entry point statement in <code>docker-compose.yml</code> file:
 entrypoint: mysqld_safe --skip-grant-tables --user=mysql
More elaborate content can be found here: [https://gist.github.com/rordi/ab55c92848fd06884172ed22ae88d7df Change root password in MariaDB Docker container running with docker-compose].

==Restoring Binary Files==
The process of restoring binary files, such as images, PDF, and other binary format data, should refer to [[Restore.sh]]. 

===Instruction for restoring Binary Files to MediaWiki===

Loading binary files to MediaWiki, one must use a maintenance script in the /maintenance directory. This is the command line information. It needs to be launched in the container that runs MediaWiki instance.

Load images from the UploadedFiles location. In most cases, the variable <code>$ResourceBasePath</code> string can be replaced by <code>/var/www/html</code>.
 cd $ResourceBasePath
 php $ResourceBasePath/maintenance/importImages.php $ResourceBasePath/images/UploadedFiles/

After all files are uploaded, one should try to run a maintenance scrip on the server that serves Mediawiki service:
 php $ResourceBasePath/maintenance/rebuildImages.php
For more information, please refer to MediaWiki's documentation on [[MW:Manual:rebuildImages.php|Manual:rebuildImages.php]].

=Backup and Restore Kubernetes=
Please refer to '''Cloud Native DevOps with Kubernetes'''<ref>{{:BOOK/Cloud Native DevOps with Kubernetes/Backing Up Cluster State}}</ref>. Particularly looking at [[Velero]].

=References=
<references />