Maintenance script "rebuild­Data.php"

From semantic-mediawiki.org
rebuildData.php
Allows to rebuild all the semantic data for a selected data backend/store
Further Information
Provided by: Semantic MediaWiki
Added: 1.9.2
Removed: still in use
Location (path): ./extensions/SemanticMediaWiki/maintenance/
Table of Contents

The "rebuildData.php" maintenance script recreates all the semantic data in the database, by cycling through all the pages that might have semantic data, and calling functions that re-save semantic data for each one, i.e. doing a full re-parse.

This script is a command line tool, while special page "SemanticMediaWiki"Provides functions for wiki administrators that assist in doing their administration work (formerly known as special page "SMWAdmin") data rebuilding (repair) uses the job queue to process all pages. If possible use this maintenance script for data rebuilding. See also the help page on using special page "SemanticMediaWiki" for data rebuilding.

Semantic MediaWiki 3.2.0Released on 7 September 2020 and compatible with MW 1.31.0 - 1.35.x. brought an improved client output to this maintenance script.1

This maintenance script deprecated the former "SMW_refreshData.php" script starting with Semantic MediaWiki 1.9.2Released on 18 April 2014 and compatible with MW 1.19.0 - 1.22.x. which was removed with the release of Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x. in October 2018. When upgrading to Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x. the migration to this script must be done.

Usage[edit]

php rebuildData.php [-d|-s|-e|-f|-n|--startidfile|-b|-v|-c|-p|-t|--namespace|--page|--redirects|--query|-f|--no-cache|--report-runtime|--debug|--skip-properties|--shallow-update|--ignore-exceptions|--exception-log|--with-maintenance-log|--revision-mode|--force-update|--dispose-outdated]
This only shows the script specific parameters.

Parameters[edit]

Maintenance scripts provide generic maintenance parameters, script dependent parameters and depending on the maintenance script script specific parameters which are described on this page if provided.

Script specific parameters
Parameter Description
-d <delay> Wait for this many milliseconds after processing an article. Useful for limiting server load.
-s <startid> Start refreshing at given object ID. Useful for partial refreshing.
-e <endid> Stop refreshing at given object ID. Useful for partial refreshing.
-n <numids> Stop refreshing after processing a given number of IDs. Useful for partial refreshing.
--startidfile <startidfile> Read <startid> from a file instead of the arguments and write the next ID to the file when finished. Useful for continual partial refreshing from cron.
-b <backend> Execute the operation for the storage backend of the given name (default is to use the current data backend/store)
-v Be verbose about the progress.
-c or --categories Will refresh only category pages (and other explicitly named categories).
NoteNote: The --categories option is only available starting with Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x..2
-p Will refresh only property pages (and other explicitly named namespaces)
--namespace Only refresh pages in the selected namespace identified by its constant, e.g. --namespace="NS_MAIN". Available since Semantic MediaWiki 3.1.0Released on 23 September 2019 and compatible with MW 1.31.0 - 1.33.x..3
-t Will refresh only type pages (and other explicitly named namespaces)
NoteNote: This option was removed with Semantic MediaWiki 2.3.0Released on 29 October 2015 and compatible with MW 1.19.0 - 1.25.x. since namespace "Type" is no longer used. 4
--page=<pagelist> Will refresh only the pages of the given names, with | used as a separator.
NoteNote: The options -s, -e, -n, --startidfile, -c, -p, -t are ignored if --page is given.
--redirects Will refresh only the pages which are redirecting to another page. Available since Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x..2
--query Will refresh only pages returned by a given query. Available since Semantic MediaWiki 1.9.2Released on 18 April 2014 and compatible with MW 1.19.0 - 1.22.x..5
NoteNote: The options -s, -e, -n, --startidfile, -c, -p, -t are ignored if --query is given.
-f Fully delete all content instead of just refreshing relevant entries. This will also rebuild the whole storage structure. May leave the wiki temporarily incomplete.
--no-cache Sets the $wgMainCacheType to none while running the script. Available since Semantic MediaWiki 2.2.0Released on 9 May 2015 and compatible with MW 1.19.0 - 1.24.x.6 with improvements in Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x.7
--report-runtime Will return memory usage and runtime of the respective script execution. Available since Semantic MediaWiki 2.1.0Released on 19 January 2015 and compatible with MW 1.19.0 - 1.24.x. as --runtime.8
NoteNote: Since Semantic MediaWiki 2.2.0Released on 9 May 2015 and compatible with MW 1.19.0 - 1.24.x. this parameter was renamed to --report-runtime.9
--debug Sets global variables to support debug ouput while running. Available since Semantic MediaWiki 2.2.0Released on 9 May 2015 and compatible with MW 1.19.0 - 1.24.x..10
--skip-properties Is to skip the default properties rebuild (only recommended when successive build steps are used). Available since Semantic MediaWiki 2.3.0Released on 29 October 2015 and compatible with MW 1.19.0 - 1.25.x..11 From Semantic MediaWiki 2.2.0Released on 9 May 2015 and compatible with MW 1.19.0 - 1.24.x. to Semantic MediaWiki 2.2.3Released on 11 October 2015 and compatible with MW 1.19.0 - 1.25.x. it was not possible to avoid the properties being rebuild first.12
--shallow-update As option is to parse only those entities that have a different last modified timestamp compared to that of its last revision and should only be used to run a quick update on deleted, redirects, and other out of sync entities. Available since Semantic MediaWiki 2.3.0Released on 29 October 2015 and compatible with MW 1.19.0 - 1.25.x..4
--ignore-exceptions Allows to ignore encountered exceptions, i.e. the script does not stop as soon as an exception (error) appears. Available since Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x..2
NoteNote: This option is best used together with the --exception-log option.
--exception-log="/path/to/smw/logs/directory/" Writes exceptions (errors) encountered to a log file allowing for later debugging. Available since Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x..13
NoteNote: A file name is automatically being created containing the string "logrebuilddata-exceptions" and the timestamp (ISO format), e.g. "logrebuilddata-exceptions-2016-12-05.log". In case an unambiguous name is needed just add an identifier to the option, e.g. --exception-log="/path/to/smw/logs/directory/mywiki-". It will be prepended to the file name, e.g. "mywiki-logrebuilddata-exceptions-2016-12-05.log".
--with-maintenance-log Adds a log entry to "Special:Logs" on the wiki and reports the script's runtime. Available since Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x..13

NoteNote:  If you are using this parameter make sure that MediaWiki's configuration parameter $wgMaxNameChars (MediaWiki.org) is set to a value not lower than "17".14 Otherwise an exception will be issued informing about the minimum value for this setting ("32" or higher is recommended).15

--revision-mode Use the revision information and compares the latest title/page revision ID with that of the associated revision ID in SMW hereby allowing to make some assumptions about the content state including:
  • If both revision IDs match then it is assumed that no content divergence occurred.
  • The wiki content (including those embedded annotations) should match on what is stored in SMW for that particular entity.
  • As a result, further processing (especially the parsing of content which is the most costly operation during a data rebuild) of that entity is skipped. Available since Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x..16
--force-update To "force" an update under any circumstance. Available since Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x..16
--dispose-outdated To dispose of outdated entities without starting a data rebuild an update under any circumstance. Available since Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x..17

NoteNote:  Since Semantic MediaWiki 3.2.0Released on 7 September 2020 and compatible with MW 1.31.0 - 1.35.x.18 one can use maintenance script "disposeOutdatedEntities.php"Allows to dispose outdated entities for this task.

--auto-recovery Allows to restart from a canceled (or aborted) index run. Available since Semantic MediaWiki 3.1.0Released on 23 September 2019 and compatible with MW 1.31.0 - 1.33.x..

Progress display[edit]

The progress (starting with Semantic MediaWiki 2.3.0Released on 29 October 2015 and compatible with MW 1.19.0 - 1.25.x.) that is displayed during a rebuild process is self-adjusting based in the amount of expected ID's vs. the actual amount of ID's being processed.19 Due to each entity (i.e. subobject, property, and subject) being assigned an ID it does not necessarily correspond to the page ID of MediaWiki as various types of subobjects embedded in a page are assigned an ID as well.

Especially in case of a full rebuild (-f) is the progress slanting where the start amount is lower than the final ID count (which is predicted from the MediaWiki articles count).

Quick and slow progress[edit]

ID's assigned to a "real" page are parsed using MW's Parser to ensure that all data and extensions influencing the state of the data are being accounted for which amounts to the extensive memory and time effort required to finalize a full parse of a page including all #subobject, #ask plus any other embedded parser function calls.20

ID's that represent data items such as subobjects or value objects can be processed using Semantic MediaWiki internal functions hence the comparatively quick update progress.

Verbose output[edit]

The verbose output (-v) got extended2 in Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x. to display additional information about an entity that is being processed. The marker * identifies a regular MediaWiki page with the ID corresponding to the page table entry while non-marked ID's are matched to an entry in the smw object ids database table.

Marked for deletion entries[edit]

Starting with Semantic MediaWiki 2.3.0Released on 29 October 2015 and compatible with MW 1.19.0 - 1.25.x., entities marked as deleted :smw-delete are being removed at each "rebuildData.php" run to free tables of outdated entities.11

The following command quietly removes just the outdated entities21
php rebuildData.php --skip-properties -s 1 -e 1 --quiet

Since Semantic MediaWiki 3.0.0Released on 11 October 2018 and compatible with MW 1.27.0 - 1.31.x. a dedicated flag is available:17

php rebuildData.php --skip-properties --dispose-outdated --quiet

Starting with Semantic MediaWiki 3.2.0Released on 7 September 2020 and compatible with MW 1.31.0 - 1.35.x. the dedicated maintenance script "disposeOutdatedEntities.php"Allows to dispose outdated entities is available.18

Dispose of outdated object ID references[edit]

Starting with Semantic MediaWiki 2.4.0Released on 9 July 2016 and compatible with MW 1.19.0 - 1.27.x. outdated object ID references are disposed when running "rebuildData.php".22 When the data type of a property type is changed, a property is removed or other object values are deleted chances are that some ID's remain in the smw object ids database table.23 To avoid a pile of garbage references being collected in this database table it is checked if for the ID's whether they can safely be removed or not during the "rebuidData.php" run. This is best and frequently done using the --shallow-update option.21.

The following command removes outdated object ID references
php rebuildData.php --shallow-update

Examples[edit]

The following command refreshes existing semantic data items with a delay of 50 ms between every data item without prompting progress information.
php rebuildData.php -d 50 -q
The following command verbosely rebuilds semantic data after deleting existing items with a delay of 100 ms between every data item.
php rebuildData.php -f -d 100 -v
The following command verbosely rebuilds semantic data of pages in a given category.
php rebuildData.php --query='[[Category:SomeCategory]]' -v
The following command verbosely rebuilds semantic data with a delay of 75 ms between every data item and provides memory usage information after it has been completed.
php rebuildData.php -d 75 --report-runtime
Example output:
Memory used: 25543928 (b: 11429464, a: 36973392) with a runtime of 81.62 sec (1.36 min)
a) memory used after execution and b) memory used before the execution
The following command refreshes the wiki pages "Page 1" and "Page 2" without prompting progress information.
php rebuildData.php --page="Page 1|Page 2"
The following command rebuilds semantic data with a delay of 50 ms between every data item, ignores errors which may arise during execution and writes them to a file in the directory provided.
php rebuildData.php -d 50 --ignore-exceptions --exception-log="/var/log/mediawiki/"
Exceptions are e.g written to the "mywiki.logrebuilddata-exceptions-2016-08-14.log" file if the wiki ID was "mywiki" and the script was run on August 14, 2016.
The following command removes outdated object ID references and adds an maintenance log entry to special page "Log" (Semantic MediaWiki log)13
php rebuildData.php --shallow-update --with-maintenance-log

Note[edit]

There was some discussion on the mailing list about the occasions it is required to run this maintenance script.24

See also[edit]











References

  1. ^  Semantic MediaWiki: GitHub pull request gh:smw:4517
  2. a b c d  Semantic MediaWiki: GitHub pull request gh:smw:1433
  3. ^  Semantic MediaWiki: GitHub pull request gh:smw:3960
  4. a b  Semantic MediaWiki: GitHub pull request gh:smw:1127
  5. ^  Semantic MediaWiki: GitHub pull request gh:smw:243
  6. ^  Semantic MediaWiki: GitHub issue gh:smw:749
  7. ^  Semantic MediaWiki: GitHub pull request gh:smw:820
  8. ^  Semantic MediaWiki: GitHub issue gh:smw:643
  9. ^  
    GitHub commit gh:smw:68e8bc9
  10. ^  Semantic MediaWiki: GitHub issue gh:smw:766
  11. a b  Semantic MediaWiki: GitHub pull request gh:smw:1106
  12. ^  Semantic MediaWiki: GitHub issue gh:smw:877
  13. a b c  Semantic MediaWiki: GitHub pull request gh:smw:1361
  14. ^  Semantic MediaWiki: GitHub issue gh:smw:1983
  15. ^  Semantic MediaWiki: GitHub pull request gh:smw:1985
  16. a b  Semantic MediaWiki: GitHub pull request gh:smw:3441
  17. a b  Semantic MediaWiki: GitHub pull request gh:smw:3284
  18. a b  Semantic MediaWiki: GitHub pull request gh:smw:4484
  19. ^  Semantic MediaWiki: GitHub pull request gh:smw:1042
  20. ^  Semantic MediaWiki: GitHub issue gh:smw:1698
  21. a b  Semantic MediaWiki: GitHub issue comment gh:smw:1754:236913464
  22. ^  Semantic MediaWiki: GitHub pull request gh:smw:1216
  23. ^  Semantic MediaWiki: GitHub issue gh:smw:498
  24. ^  Semantic MediaWiki: User mailing list thread "When is it required to run rebuildData.php"