This section explains the use of the server executables, configuration files and general administration procedures. These include backup, server tuning and profiling, installing extension packages etc. See the Access Interfaces chapter for a discussion of the command line SQL interface.
A typical installation will require a minimum of 400Mb of hard disk space to install the code, samples, documentation and sample database. The database will need additional space for data inserted, backups, logs and reports, web pages, etc.
The size of the database .db file will not reduce when data is removed. The spare space will however be reclaimed for later use.
The Virtuoso database requires a minimum of 64Mb of system memory for each instance to operate in. Each connection will take between 70kb and 130kb of memory.
The memory usage is affected by the following parameters in the Virtuoso configuration file.
See the following Parameter section for more details.
Virtuoso runs on the following operating systems:
The below table lists the most important parameters and limits for a database or a Virtuoso server instance. For space consumption of individual data types, see Space Consumption.
Multiple Virtuoso server instances may coexist on a single machine. Each database will need to be assigned a unique TCP port number. On the windows platforms (except 95,98,ME) the Virtuoso can be configured for multiple services. For further details, see the Creating and Deleting Virtuoso Services in the Installation chapter.
To run Virtuoso on a machine, only the Virtuoso server executable and a valid virtuoso.ini file are needed. An empty database file will be created automatically at first startup. None of the other files in the installation are needed for basic operation. Client interfaces may additionally need ODBC drivers or the like to be set up in system configuration files or the Windows registry but the server itself does not depend on these.
Virtuoso comes with optionally installable SQL application packages (VAD packages) for web based admin, on-line documentation, programming tutorials and a BPEL processor. The installer typically asks whether to install these into the demo or the default empty database. Depending on the OS and form of installer, also depending on whether the installation is the commercial or open source one, the package files will be in different locations. To locate them, look for *.vad. The packages are typically made in two variants, where one keeps the installed items in the DAV repository and the other in the file system. They are functionally equivalent in most cases, except for some tutorials that will only work in the file system. Keeping installed resources in DAV has advantages when moving the database or backing up, since the installed items will not have to be treated separately. To install these on a database, login as dba using the isql utility and issue the SQL command:
# at the OS command line prompt, assuming the dba password is dba and the server is at the default port 1111: $ isql 1111 dba dba -- At the SQL prompt: SQL> vad_install ('conductor_dav.vad', 0); -- The path is relative to the server's working directory.
Since the packages are read from and sometimes the extracted contents are written to the file system, the server must have access to the directories in question. Checjk OS permissions and set the DirsAllowed parameter in the ini file to allow the needed access.
After the packages are installed, they can be used by pointing the web browser to their start page. The start pages are listed on the root page of the demo database and at the Getting Started section of the Open Virtuoso web site. The start page for the Conductor web admin interface is /conductor, the docs are at /doc/html, the tutorials at /tutorial, the BPEL admin and demos at /BPELGUI. All these are directly under the default Virtuoso web listener, the port is shown in the messages log and read from the HTTP Server section of the virtuoso.ini file.
The Virtuoso server requires a valid license file before it will successfully start. The license file is a file always called virtuoso.lic and must reside in the working directory of the database instance - where the <database-name>.db file resides for the instance.
Evaluations versions of Virtuoso come without a license file, however the registration procedure takes your email address, which will be used to email a new license file for you evaluation.
Virtuoso provides extensive log information for resolving problems. Should any problems arise Virtuoso logs should always be consulted.
The Virtuoso server by default will send log information to two places, the appropriate system log for the operating system and the virtuoso.log file. On Unix based operating systems, including Linux, this information will appear in the system log files. On Windows the Virtuoso log information will appear in the Application Event Log.
Virtuoso logs information in the system logs before the Virtuoso.log file is open. This advantageously will log errors that cannot be placed in the virtuoso.log file, such as when the virtuoso.ini file cannot be located during Virtuoso startup.
The system log feature can be disabled using the "Syslog" parameter in the [Database] section. This is described in more detail in the following section.
The virtuoso.ini file is read from the directory that is current at server startup. This file contains an entry for all user settable options in the server. It is divided into the following sections:
[Database] Location of database files
[Parameters] Server tuning, options
[HTTPServer] Settings in this section control the web server component of the Virtuoso Server.
[URIQA] Settings in this optional section control URIQA semantic web enabler.
[SPARQL] Settings in this optional section control SPARQL protocol endpoint default parameters and limits.
[I18N] encoding settings
[Replication] The replication section sets the transactional replication parameters for the server.
[Mono] The Mono section contains settings for Virtuoso with Mono Runtime CLR support.
[Client] Client default settings
[AutoRepair] Corrupted database recovery
[Striping] Multi-file / multi-disk databases
[VDB] Virtual database functionality related options
[Ucms] location of UCM files describing multi-byte encodings such as Far East languages
[Zero Config] ZeroConfig related options
[Plugins] VSEI plugin modules
Below are the descriptions for each parameter
For a single file database, this is the relative path of the file in the format appropriate to the platform. The path is resolved relative to the directory that is current at server startup. All other paths are interpreted similarly.
This is the transaction log file. If this parameter is omitted, which should never be the case in practice, the database will run without log, meaning that it cannot recover transactions committed since last checkpoint if it should abnormally terminate. There is always a single log file for one server. The file grows as transactions get committed until a checkpoint is reached at which time the transactions are applied to the database file and the trx file is reclaimed, unless CheckpointAuditTrail is enabled.
This file logs database error messages, e.g. 'out of disk'. By viewing this the dba can trace problems and see at which times the server has started, checkpoints have been made, etc.
This controls what events get logged into the database error log. This should always be 7.
This optional parameter can be used to manually specify the location of the Virtuoso lock (.lck) file. This can be relative or the full path to the lock file. Virtuoso, by default, creates a file with the same name as the DatabaseFile but with the extension of .lck. This file exists when the Virtuoso server is running to prevent it starting multiple times using the same parameters, and should be automatically removed by the server upon exit. However, not all file systems support file locking, such as NFS, therefore this parameter can be set to keep the lock file on a more appropriate file system.
This is the size that the database file automatically grows (in 8k pages) when the current file is not large enough. Default = minimum = 100. The parameter has no effect if striping is set.
A non-zero value will enable the settings in [Striping] to take effect. If this is the case the DatabaseFile parameter is ignored and the files in [striping] are used.
If this is non-zero log segmentation is enabled. This is only used for crash dumps where several files may be needed to accommodate the recovery logs. If non-zero, this will be followed by entries of the form Log1=...
The number in Log<n> is the ordinal number of the log, starting at 1. The entry consists of the file name and allocated size with an optional size modifier. Available modifiers are B for blocks of 4k (default if unspecified, K for kilobytes, M for megabytes, and G for gigabytes. The log is filled up to the first transaction that exceeds the size. The log therefore will be longer than the allocated size. If blobs are involved, this amount can be quite substantial. It is therefore advisable to have some extra space available on the storage device used for recovery logs.
These options make it possible to produce a crash dump of a specified range of disk pages. In case the server runs out of disk space during a dump, the error message indicates the page at which the next dump should be started after the user has made more space available. In this case it is important to rename the already produced crash dump or change the Logx= -entries in virtuoso.ini to avoid overwriting said files. If none of these options are provided, the server will attempt a crash dump of the entire database storage.
The name of a section in the INI file containing temporary database details. If this parameter is omitted or the section does not exist the default values for temporary storage are used.
Virtuoso normally writes log worthy messages to the system log (Unix based operating systems including Linux) or the Windows Event Log (Windows operating systems). Messages are written in the system/event log before the virtuoso.log file is opened, therefore errors due to absence of virtuoso.ini log are loggable there. This system/event logging can be disabled using this option, by default it is set to 1 meaning on.
On Unix/Linux messages are written as "Virtuoso" events.
On Windows messages are written in the Application event log.
This section name must match the TempStorage parameter in the Database section of the Virtuoso INI file to be of any use. Otherwise this section will be ignored.
Name of temporary database file.
Name of temporary transaction file.
Increment amount by which the database file will dynamically grow. This setting is identical in use and purpose to parameter in the Database section with the same name.
This is a Win32 specific option that forces Virtuoso to only run on one CPU in a multiprocessor environment.
This is the IP Address and port number where the server will start listening. You do not need to specify the listening IP Address but can do in a situation that you want the server to bind to a specific address only.
This is the maximum number of threads used in the server. This should be close to the number of concurrent connections if heavy usage is expected. Different systems have different limitations on threads per process but a value of 100 should work in most places.
Stack size of thread used for reading client messages and accepting connections.(default : 50 000 bytes)
Stack size of the main thread (default : 100 000 bytes)
The interval in minutes (default : 0) after which threads in the thread pool should be released.
The maximum number of threads (default : 10) to leave in the thread queue after thread clean-up interval has expired.
The interval in minutes (default : 0) after which allocated resources will be flushed..
Stack size of worker threads. This is the stack size for serving any client SQL statements or HTTP requests. This can be increased if the application uses recursive stored procedures or links in external code needing a large stack. (default 100 000 bytes)
A Percentage that may be greater than 100%. This gives a percentage of the main .db file to which the temp db file may grow before starting to reclaim the oldest persistent hash index. Basically if a particular hash index is reusable (i.e. it references only table columns and the values in these columns have not changed) the engine keeps the hash index defined into the temp db for reuse. This parameter allows some control over the temp db file size.
If this is non-zero, the database file(s) will be opened with the O_DIRECT option on platforms where this is supported. This has the effect of doing file system I/O from application buffers directly, bypassing caching by the operating system. This may be useful if a large fraction of RAM is configured as database buffers. If this is on, the file system cache will not grow at the expense of the database process, for example it is less likely to swap out memory that Virtuoso uses for its own database buffers. Mileage will vary according to operating system and version. For large databases where most of system memory is used for database buffers it is advisable to try this.
This is the interval in minutes at which Virtuoso will automatically make a database checkpoint. The automatic checkpoint will not be made if there is less than MinAutoCheckpointSize bytes in the current transaction log. A checkpoint interval of 0 means that no periodic automatic checkpoints are made.
Setting the value to -1 disables also the checkpoints made after a roll forward upon database startup. This setting should be used when backing up the database file(s). This guarantees that even if the server dies and restarts during the copy, no checkpoints that would change these files will take place and thus the backup is clean.
the checkpoint_interval SQL function may be used to change the checkpoint interval value while the database is running.
This controls how the file system is synchronized after a checkpoint. Once the checkpoint has issued all write system calls it needs it can do one of the following depending on this setting:
0. - Continue, leave the OS to flush buffers when it will.
1. - Initiate the flush but do not wait for it to complete.
2. - Block until the OS has flushed all writes pertaining to any of the database files.
This controls the amount of RAM used by Virtuoso to cache database files. This has a critical performance impact and thus the value should be fairly high for large databases. Exceeding physical memory in this setting will have a significant negative impact. For a database-only server about 65% of available RAM could be configured for database buffers.
Each buffer caches one 8K page of data and occupies approximately 8700 bytes of memory.
Specifies how many pages Virtuoso is allowed to remap. Remapping means that pages can consume the space of two actual pages until checkpoint. See the Checkpoint & Page Remapping section in the SQL Reference chapter for more information.
This setting should always be 0.
This controls the case sensitivity of the Virtuoso SQL interpreter. The following values are supported:
0 - SQL is case sensitive and identifiers are stored in the case they are entered in. This is similar to the Progress or Informix default.
1 - SQL is case sensitive and Unquoted identifiers are converted to upper case when read. To use non-upper case identifiers, the identifiers must be quoted with double quotes ("). This is similar to Oracle.
2 - SQL is case-insensitive and identifiers are stored in the case where first seen. Unlike the situation in other modes, two identifiers differing only in case cannot exist. This is similar to the MS SQL Server 7 default behavior.
Once a database is created in one case mode the case mode should not be changed as that may change the interpretation of stored procedures etc.
See CheckpointInterval.
This is the size of transaction log in bytes after which an automatic checkpoint is initiated. If this is non-zero, whenever the transaction log size exceeds this size an automatic checkpoint is started. This will result in approximately like sized logs to be generated. This is useful with the CheckpointAuditTrail option for generating a trail of equal sized consecutive logs.
If this is non-zero each checkpoint will start a new log and leave the old transaction log untouched. A 0 value indicates that the transaction log may be reused once the transactions in it have been written in a checkpoint.
If it is important to keep an audit trail of all transactions in the database's life time then this option should be true. A new log file will be generated in the old log file's directory with a name ending with the date and time of the new log file's creation. Old log files can be manually copied into backup storage or deleted. Only one log file is active at one time. The newest log file may at any time be written to by the server, but that is the only log file Virtuoso has open at each time. Thus reading any logs is safe. Writing or deleting the active log file will result in loss of data, and possibly referential integrity, in the database. See the Back and Recovery chapter for more information on this and related parameters.
If non-zero the system SQL function is enabled. This will allow a dba group user to run shell commands through SQL. This poses a potential security risk and hence the setting should normally be 0.
This is the maximum number of rows returned by a static cursor. Default = 5000
This is the amount of text data processed in one batch of the free-text index when doing a batch update or non-incrementally reindexing the data. Default : 10,000,000
When set to 1, if an application prepares a statement with insufficient number of input parameters, the unspecified ones are assumed to be NULL.
Defines a sorting order according to SYS_COLLATIONS. The name supplied to this parameter must be in charsets_list(1).
<path> := <absolute_path> or <relative_path> comma-delimited list of OS directories allowed for file operations such as file_to_string(). The server base directory (the directory containing this INI file) must appear on this list in order to enable File DSNs to work. On Windows use in the path "\".
The Virtuoso ISQL utility can be used to check the Server DirsAllowed params as follows:
SQL> select server_root (), virtuoso_ini_path ();
The above should show in the result the server working directory and INI file name.
Also you can check the relevant INI setting by running following statement via ISQL command line utility:
SQL> select cfg_item_value (virtuoso_ini_path (), 'Parameters', 'DirsAllowed');
<path> := <absolute_path> or <relative_path> OS directories denied for file operations. See Virtuoso ACL's for information on functions that are restricted.
Specifies the port on which the server listens for incoming SSL CLI requests.
The SSL certificate to use (same meaning as the SSLCertificate in HTTPServer section)
The server's private key (same meaning as the SSLCertificate in HTTPServer section)
This parameter governs the maximum number of partial or full join orders that the Virtuoso SQL Optimized compiler will calculate per select statement. When MaxOptimizeLayouts has been reached, the best execution plan reached thus far will be used. The default value is 1000, specifying 0 will try all possible orders and guarantee that the best plan is reached.
If non-zero, this specifies that the SQL compiler should stop considering alternative execution plans after the elapsed compilation time exceeds the best run time estimate times the parameter. For example, if this is 2, then compilation stops after using twice the time of the best plan reached thus far. Enabling this option increases performance when processing short running queries that are each executed once. Using this with long running queries or prepared parametrized queries is not useful and may lead to non-optimal plans being selected.
This parameter accepts a comma-delimited list of tracing options to activate by default. Enabled trace options will list there respective errors in the virtuoso.log file when encountered. Valid options are:
If an invalid option is set then this error will be listed in the virtuoso.log file upon server startup. Virtuoso will continue to log selected options unless the trace_off() function is called for that item.
The functions: trace_on() and trace_off() which use the same options and can turn logging options on/off while the server is running.
[Parameters] .... TraceOn = soap, errors ...
This will enable logging of additional information regarding SOAP calls and SQL run-time errors into the virtuoso.log file.
[Parameters] .... ThreadCleanupInterval = 1 ResourcesCleanupInterval = 1 ...
Set both to 1 in order to reduce memory leaking.
Determines whether Virtuoso encryption should be accepted from client connections. The default value is 1 - on. If set to 0 then only clear text and digest authentication will be accepted.
This parameter is applied to the environment prior to the server's startup. It is valid only for binaries hosted in the Virtuoso Java VM. This has the same format as the Java CLASSPATH environment variable for the platform being used.
Virtuoso searches for classes in the following order:
These can be used for setting Java options for the Java runtime hosted in Virtuoso. These options work as if provided as command line options to the JRE's Java command line.
More than one line of options can be specified by using consecutively numbered options:
JavaVMOption1 = -Ddt1=val1 JavaVMOption2 = -Ddt2=val2 ... JavaVMOption5 = -Ddt5=val5
The PLDebug switch controls the type of debugging enabled:
Controls whether Virtuoso will report call stack on errors.
This parameters takes the following values:
This controls whether Virtuoso will recompile all stored procedures listed in SYS_PROCEDURES and internal PL procedures defined during server startup. By default Virtuoso will recompile all procedures, with this setting set to 0 Virtuoso will defer compilation until procedures' first call. The benefits of this are that Virtuoso may start up up to 2-3 times faster, also the initial memory consumption will be significantly reduced as it does not need to analyze all the long varchar data allocating memory for execution. This setting is a boolean, either 1 or 0. This setting does not apply to attached VDB procedures or modules.
Controls the number of file descriptors per file to be obtained from the OS. The default and minimum value is 1. This parameter only effects databases that use striping. Having multiple FDs per file means that as many concurrent I/O operations may simultaneously be pending per file. This allows more flexibility for the OS to schedule the operations, potentially improving file I/O throughput
This option controls the behavior of free-text triggers in super-tables. If this option is set to 1 then Virtuoso will scan the hierarchy of tables until a free-text index is used to use when compiling SQL statements involving contains, xcontains or xpath_contains.
This option controls the behavior of super-table triggers. When this option is set to 1 then triggers in the super-table will be called before its own (sub-table) triggers. This behavior is recursive and will continue up the branch of recursion, hence the triggers in the top most table in the chain will be called first.
The TOP select statement clause caches in memory the rows pertinent to the result. The number of rows allowed to be cached within memory is limited by this parameter.
Simple example using OFFSET and LIMIT:
Virtuoso uses a zero index in the OFFSET. Thus in the example below, will be taken position at record 9000 in the result set, and will get the next 1000 rows starting from 9001 record. Note that the MaxSortedTopRows in parameters Virtuoso ini section needs to be increased (default is 10000).
select ?name ORDER BY ?name OFFSET 9000 LIMIT 1000
This parameter is only applicable to Unix servers. Virtuoso clients on the localhost of the server can benefit from using Unix Domain sockets to improve connection performance. By default (DisableUnixSocket = 0) Virtuoso will open a Unix Domain listen socket in addition to the TCP listen socket. The name of the UD socket is /tmp/virt-<tcp-listen-port>. When a client attempts to connect to the Virtuoso server using the specific address localhost it will first try connecting to the UD socket, failing that it will silently revert to the TCP socket. See the Unix Domain Socket Connections section for more details.
When the roll-forward log entry of a transaction exceeds this size, the transaction is too large and is marked as uncommittable. This work as upper limit otherwise infinite (transactions). The default is 50Mb . Then also note that transaction roll-back data takes about 2x of roll-forward data. Hence when the transaction roll-forward data which is 50Mb the total transient consumption is closer to 150 Mb.
Directory for storing temporary data for large object handled in replication and HTTP server. Defaults to server home directory.
Enable or disable Database Event Hooks.
Specifies the OS user name to which the server will switch after opening the listen ports. Has an effect only on the operating systems that support it.
Specifies the limit of the memory to be used for compiling a SQL statement. If the query compilation requires more memory an error will be signalled. If this is a zero then no limit will be applied. The default is 10Mb i.e. when no parameter is specified, also if negative number or less than 5000000 is given then it would be set to 5000000 bytes.
This specifies the default transaction isolation. This isolation is used unless overridden by a client setting it using the SQL_TXN_ISOLATION or equivalent connection option or a stored procedure locally setting it with the SET statement. The values are as by the SQL_TXN_ constants in ODBC, that is, 1 for read uncommitted, 2 for read committed, 4 for repeatable read and 8 for serializable. If nothing is specified, the default is repeatable read.
This specifies whether to use asynchronous file I/O on supporting Unix systems. A value of 0 means not using it. A value of 1 means using lio_listio for any background write or read ahead if available. A value of 2 means to use the regular blocking read and write but to merge adjacent operations into a single system call when possible.
Controls the acceptable size of the temp database file. If on startup it's size (in MB) is greater than TempDBSize the file gets deleted and reset. This feature can be turned off by setting TempDBSize to 0. Note that the temp db file serves as an optimization storage only and doesn't have any client data that are not in either the main database files or the corresponding transaction log files.
Runs server in lite mode. When Lite mode is on:
The size of hash to control rdf free text index
Size of index tree maps, larger is better for speed but consume memory, in lite is 2 in 'normal' mode is 256 by default.
Disable odbc listen on tcp port, if this given unix socket will always be used. In lite mode is 0, thus no tcp listen for sql/odbc connections, can connect only via unix socket.
Settings in this section control the web server component of the Virtuoso Server.
This specifies the initial HTTP listen port for the HTTP server. Can be specified also as ipaddress:port. Once Virtuoso is started it is possible to create multiple listeners using virtual directories.
The stack size of the HTTP thread used for reading/processing HTTP client requests and accepting connections. The default is 120,000 bytes. This parameter cannot have value less than the default; if a smaller value is specified the default will be used.
This specifies the number of concurrently serviced HTTP requests. If there are more concurrent requests, accepting the connections will be deferred until there is a thread ready to serve each. If an attempt to exceed the number of licensed connections is found. the latter will be used. When the number of threads is low some of the tutorials may perform poorly and appear not to work; this is due to the demonstration licensing. When testing Virtuoso with the demonstration license it can be quite easy to hit the limits unless remote connections and HTTP connections are conserved. You may wish to either wait for previous transactions to finish or restart the server to be sure.
The stack size of HTTP thread used for reading/processing HTTP client requests and accepting connections. The default setting, if not supplied, is 120,000 bytes. The default value is the minimum; lesser values will be rounded up.
This is the file system path of the root directory of files served by the Virtuoso web server. The index.html in that directory will be served for the / URI. If relative, the path is interpreted relative to the server's working directory.
String passed as Server: header to HTTP client. This string is not required, in its absence the above default will be assumed. This string can be set to anything required.
String passed as User-Agent: header to server by HTTP client. This string is not required, in its absence the above default will be assumed. This string can be set to anything required.
Allows you to set the default server character set. If no default is specified then ISO-8859-1 will be used automatically by the server.
This sets the default behavior of HTTP transmission. If set to 1 The Virtuoso HTTP server will send GZipped content to user agents. Otherwise content will be sent as is. The function http_enable_gz() lets you change the server mode on the fly.
Connections by HTTP 1.1 clients can remain open after the initial response has been sent. This parameters sets a cap on how many socket descriptors will at most be taken by keep alive connections. Such connections will be dropped by the server ahead of timeout if this number would be exceeded. Thus the maximum number of open sockets for the Virtuoso HTTP server is this number plus the number of threads. A keep alive connection is by definition not associated to any pending processing on any thread.
This is a timeout in seconds before Virtuoso closes an idle HTTP 1.1 connection.
Setting this to 1 activates Virtuoso proxy service capabilities. The default value of 0 deactivates the proxy service.
Ports on which proxying is enabled should not be presented to the outside world under any situation.
If specified, Virtuoso will produce an HTTP server log file with the date appended to the name given in the parameter and the files extension as ".out". The log file is rotated daily.
It will contain the following information:
logDDMMYYYY.out :-
An example of which is:
127.0.0.1 - - [12/Sep/2006:12:31:17 +0300] "GET /ods/ HTTP/1.1" 200 19453 "" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko.." 127.0.0.1 - - [12/Sep/2006:12:31:18 +0300] "GET /ods/default.css HTTP/1.1" 200 41389 "http://localhost:8894/ods/" "Mozilla/5.0 (Windows; U; ..." 127.0.0.1 - - [12/Sep/2006:12:31:19 +0300] "GET /ods/common.js HTTP/1.1" 200 7481 "http://localhost:8894/ods/" "Mozilla/5.0 (Windows; U; ..."
When Virtuoso is acting as a proxy or HTTP client, as is the case with the http_get or the SOAP client functions, it caches connections to HTTP 1.1 servers. This is the maximum population of said cache.
This is a timeout in seconds for dropping idle connections to other HTTP servers. These result from http_get, SOAP client functions and proxying.
This specifies the root path of DAV resources. If DAV specific HTTP methods are used on Virtuoso, these should only reference resources with paths starting with this. This is the top level DAV collection. Other paths can be declared as managed in DAV using the virtual directory mechanism, but this applies to the default virtual directory.
The Virtuoso administrator can enforce a quota on all DAV accounts, apart from the "dav" administration user, that restricts that amount of space a DAV user can consume in their DAV home directory. When this parameter is set to one (1) quotas are enabled, when this parameter is set to zero (0), they are disabled. The default quota value is five Megabytes (5MB) but can be uniquely defined for each user. Every user has a quota except "dav", which is unlimited. Dav quotas are disabled if this parameter is not specified for backwards compatibility.
Virtuoso send resources to from WebDAV to the requesting client in chunked encoding to save memory if the request is in HTTP/1.1 and the files size is greater then the DAVChunkedQuota number of bytes. The default chunked-quota value is approximately one-megabyte.
this is the port on which SSL HTTPS connections will be accepted. Can be specified also as ipaddress:port. If unspecified then the service will be disabled.
the option must point to the file with the server certificate in PEM format.
If the option begins with 'db:' e.g. 'db:id_server', then certificate and key will be loaded from DBA key space.
points to the file containing the RSA private key in PEM format.
the certificate/key pair must be valid (eq. certificate is generated on base of key)
If the option begins with 'db:' e.g. 'db:id_server', then certificate and key will be loaded from DBA key space. Note: in the case of database stored key, both SSLCertificate and SSLPrivateKey settings MUST be same.
Whether the server will require X509 certificates from the browsers.
Specifies how deep the client certificate verification process will traverse the Issuer chain before giving up. The default value of 0 means that only verification of the client certificate itself being in the server's CA list is performed. Setting this option to -1 will ignore the depth checks
a PEM file of all the X509 certificates of the Certification Authorities (CA) which the server will use for verifying the client's client certificate. A list of these will be sent to the client as a part of the SSL handshake so the client will know which certificate to send.
Note this option is in effect when X509ClientVerify is 1 or 2. If X509ClientVerify is 3 and X509ClientVerifyCAFile is given, server will send certificated during handshake, thus some clients may restrict user to send any certificate.
Defines the TCP port number on which the Virtuoso POP3 server will listen. The POP3 server will be disabled if this value is 0 or undefined.
Defines the TCP port number on which the Virtuoso NNTP server will listen. The NNTP server will be disabled if this value is 0 or undefined.
The Virtuoso FTP server can be enabled by supplying this parameter with a value. This value will then be the listening port of the FTP server. The usual port for FTP is port 21.
The Virtuoso FTP client and server use FTPServerMinFreePort and FTPServerMaxFreePort parameters as lower and upper bounds for active and passive connections. This parameters sets the lower bound.
The Virtuoso FTP client and server use FTPServerMinFreePort and FTPServerMacFreePort parameters as lower and upper bounds for active and passive connections. This parameters sets the upper bound.
If specified Virtuoso will produce an FTP server log file with the date appended to the name given in the parameter and the files extension as ".log". The log file is rotated daily.
It will contain the following information:
ftpserverDDMMYYYY.log :-
An example of which is:
hostname anonymous [22/Oct/2003:15:21:43 +0300] "PASS user@domain.com" 230 0 hostname anonymous [22/Oct/2003:15:23:11 +0300] "LIST" 226 162 hostname dav [22/Oct/2003:15:25:00 +0300] "PASS <hidden>" 230 0
Allows the FTP server to be accessible via the "anonymous" user login. The anonymous user is not a real user, it has no SQL or DAV login ability. The anonymous user can only access collections or resources that are set to public.
Default SMTP server name and port, this is used when the first parameter of the smtp_send function is omitted or NULL.
Allows you to choose what file system directory to be used for temporary storage of ASPX files hosted in DAV.
When set to "1" prevents Virtuoso from removing the plugin interpreters from the HTTP threads after each request. The default setting is "0".
The size threshold for large objects received by HTTP server. When this limit is exceeded by incoming or outgoing data, this will be stored in a temp file, see 'TempSesDir' parameter in 'Parameters' section.
The name of a HTML page or other static content to be returned to the user agents when server is running in atomic mode. Note that this page should be self-contained, i.e. no image or CSS not JavaScript references can be used.
The URIQA section sets parameters of both URIQA HTTP extension and URIQA web service. Section URIQA Semantic Web Enabler contains detailed description of this functionality, including more details about URIQA configuration parameters. This section should stay commented out as long as URIQA is not in use.
The server name, including domain and port if needed, such as "www.example.com" or "www.example.com:8088".
List of various allowed spellings of the server name, such as "www.example.com, mail.example.com, mail, localhost, localhost.localdomain".
List of various allowed spellings of the server name in form of pattern strings for SQL "like" operator, not in form of exact strings.
Do not use this without an explicit need, to let server configure itself. Please refer to detailed description before use.
If DynamicLocal is on and the host part of the IRI matches the Host header of the HTTP request in context or the DefaultHost if outside of HTTP context, then this is replaced with local: before looking up the IRI ID.
The SPARQL section sets parameters and limits for SPARQL query protocol web service service. This section should stay commented out as long as SPARQL is not in use.
Section RDF Data Access and Data Management contains detailed description of this functionality.This controls processing of the "query-uri" parameter of the SPARQL query protocol webservice, means enable 1 or prohibited 0.
Cache Expiration time in seconds that overrides Sponger's default cache invalidation.
This controls processing of the "xslt-uri" parameter of the SPARQL query protocol webservice, means enable 1 or prohibited 0.
This setting is used to limit the number of the rows in the result, if this is higher than "maxrows" parameter this setting takes precedence.
IRI of the default graph to be used if no "default-graph-uri" parameter is specified.
This setting is used to limit the estimate time cost of the query to certain number of seconds, the default is no limit.
This setting is used to set the transaction execution timeout to certain limit in number of seconds, the default is no limit.
IRI of graphs over which the sponger not to be able able to write.
IRI of notification service to which the sponger results will be send.
Default SPARQL Query.
Defer Loading of inference rules at start up.
Maximum amount of memory that is allowed for temporary storing parts of a SPARQL query result. Default is zero for no limit. ResultSetMaxRows is maximum length of a SPARQL query result. Default is zero for no limit. These two options may be useful for protecting public web service endpoints against DOS attacks, but at the same time it may cause returning incomplete results without reporting errors. When used, it is strongly advised to set the value orders of magnitude larger than the expected size of longest reply. As a rule of thumb, timeout should happen before this limit has reached. Values less than 1000000 bytes are impractical in all cases.
The SPARQL INI can be get as RDF via http://cname/sparql?ini service.
Default is 0. When 1 then file access and directory listing functions may use wide strings as file names. If a file name contains non-ASCII characters then directory listing result set will contain a wide string instead of narrow string with question marks. Note, however, that most of existing applications do not support wide strings for that purposes, so the feature should be used with extreme care.
Encoding identifier. If set, file names are translated from wide strings and default server encoding to the specified encoding before using them in file manipulation BIFs. This is useful when server's filesystem is in some national encoding or in UTF-8. The translation is bi-directional: directory listing items are translated from volume encoding to the default server encoding or wide.
Encoding identifier. If set, this encoding is used when the use of VolumeEncoding causes encoding errors. The most popular case is when an UNIX filesystem tree with UTF-8 contains some mounts of legacy storages with KOI or CP encoding. Less popular is UTF-8 volume encoding and UTF-8QR as emergency encoding to recover poorly encoded filenames.
List of directories in same syntax as DirsAllowed, default is empty. If set, all file names in the listed directories are supposed to be in VolumeEmergencyEncoding and VolumeEncoding is even not tried. The most popular case is when an UNIX filesystem tree with UTF-8 contains some mounts of legacy storages with KOI or CP encoding.
The replication section sets the transactional replication parameters for the server.
A boolean parameter controlling whether a Virtuoso can or cannot act as a transactional replication publisher.
This identifies the server instance. The entries in SYS_REPL_ACCOUNTS where SERVER equals this name are considered locally published accounts. This is the value returned by repl_this_server () SQL function.
This controls how much synchronized transactional subscribers may fall behind before being disconnected. This controls how much memory the server will use to buffer undelivered replication casts. If the queue exceeds this byte amount subscribers are disconnected and must request re-synchronization. The byte count refers to the total length of the replay records being buffered. The actual memory usage is somewhat greater.
A path to the directory where the Mono system assemblies are located. Usually it is a compile time setting, but the MONO_ROOT overrides it. This ini setting overrides the Mono environment variable of the same name.
A colon separated list of directories where the assemblies are located to be found by Assembly.Load (equivalent to MS.NET Global assembly cache). This ini setting overrides the Mono environment variable of the same name.
A path where the 'machine.config' file is to be found while running the ASPX code in Mono. This ini setting overrides an environment variable of the same name. This is also usually a Mono compile time default.
A fully qualified path and filename of the virtclr.dll virtuoso helper assembly.
Mono debug tracing can be enabled by setting this parameter to On. When tracing is On, Mono debug output with be sent to the Virtuoso debug console.
This sets the initial value of the SQL_QUERY_TIMEOUT statement option in connected clients. The ODBC standard value is 0, meaning indefinite, which is impractical in many applications. This allows overriding the default. The timeout is expressed in seconds. If the client application sets this option in a statement, this default is overridden for the statement in question.
This is an ODBC extension option allowing setting a maximum duration for a transaction. 0 means that there is no maximum.
For a forward only cursor, this option sets the number of rows prefetched at the execute and on subsequent fetch requests. A high value will speed up long selects but will be a disadvantage if only the first few rows are fetched from a cursor that has a large result set.
This should not be confused with the SQL_ROWSET_SIZE setting for scrollable cursors.
This option specifies the maximum number of bytes the server will send as prefetched rows on a forward only cursor. If long rows are being prefetched this will cut off the prefetch after this many bytes even if the number of rows is less than SQL_ROWSET_SIZE.
This options is 0 by default and can be either 1 or 0. This option controls Virtuoso's interpretation of the backslash in PL text which is normal interpreted as escaping rather than literal.
Setting SQL_UTF8_EXECS = 1 enables UTF-8 identifier storage and retrieval, whereas setting SQL_UTF8_EXECS = 0 disables it. The default setting is 0: disabled for backwards compatible. See the Wide Character Identifiers section for more information
When SQL_BINARY_TIMESTAMP is set to 1 Virtuoso will describe all TIMESTAMP columns as SQL_BINARY. If it is set to 0 then Virtuoso will report the TIMESTAMP columns as SQL_TIMESTAMP
This setting can be used to prevent SQLTables from returning system tables. The default value of this setting is 0, which is allow system tables to be returned in the normal way. Setting this to 1 will prevent system tables from being returned from SQLTables. The client can also issue a "set SQL_NO_SYSTEM_TABLE = 1/0" statement to set this in-connection.
As a result of an internal error in the database the physical integrity of references may be lost. Enabling this option causes the database to automatically repair such faults without having to go through a crash dump plus restore. A value of 0 should be normally used.
Boolean parameter which allows the Virtuoso VDB to use Array parameters if the remote data source supports these.
This parameter turns on/off MTS support in Virtuoso. It is applicable to windows multithreaded version of the Virtuoso server only.
Specifies a size of the parameter batch used by the VDB (default = 10)
The time (in seconds) after which a VDB connection is considered timed-out and closed. Default : 1000
This setting can be enabled to improve compatibility with the MS Oracle Driver which has problems with mixed case table names in catalog calls.
Boolean parameter : allows a special mode for the ORACLE ODBC drivers which do not return correct catalog data for mixed case tables. When this is on and the return from the catalog functions is empty, then a Catalog function is issued to take the same catalog data and the result is filtered on the client side.
An boolean parameter controlling whether the VDB Catalog functions called while attaching a table will be called in AutoCommit mode (required by some Sybase drivers).
The default setting of 0 instructs the VDB layer to return underlying DB errors to the client rather than automatically reconnecting without reporting the error, which would be setting 1.
The default option is 1, this forces the VDB to map a single thread to each ODBC session in the VDB rather than multiple threads. Some ODBC drivers expect that calls concerning a particular connection all take place on one thread. For example, the Oracle 8.xx ODBC driver can produce flaky crashes if this option is not set.
0 - If a second request comes while a thread is processing the epilogue or previous request for same connection the second request is scheduled for the same thread. This is called burst mode. Once this mode is active all requests on this connection will be processed in the same thread. until there is a break at delay more than 'PrpcBurstTimeoutMsecs' elapses between ending the last and receiving the next in same connection. This break in activity return the thread to the thread pool. The burst mode optimization saves thread switching time and may improve performance by up to 30% in situations involving short requests immediately following each other.
"ForcedFixedThread" mode (4). This is the same as VDBFixedThread mode, except that the thread enters this state immediately on connection (as opposed to doing that on the first VDB activity). Otherwise the scheduling stays the same as in VDB FixedThread.
"Forced Burst" mode (8). When a connection is initially made it's added to the select thread for monitoring. When an RPC comes in it causes the select thread to take the connection out of the select thread and assign that connection to designated worker thread. That thread from now on will read the data coming in from that connection directly (not depending on the select thread to detect the incoming data and wake the worker thread up). That mode saves the thread switch done handling each RPC (from the select thread to the worker thread). When the connection terminates the thread is unbound from it and returned to it's default idle state. If an lengthy RPC is being processed the server will switch the thread from burst mode to default mode while running the RPC, so the asynchronous cancellation requests can be processed.
To disable ever going to burst mode, this option can be set to 2
RPC burst mode timeout in milliseconds. (see above)
When enabled causes Virtuoso to wrap SQLAllocConnect/SQLConnect calls sequence in a mutex, thus preventing abnormal program exits when there are a large number of connection requests to the VDB. Certain ODBC drivers have been noted to produce flaky errors when a large number of threads is concurrently inside one of the ODBC connect calls..
This setting controls SQL compilation (not execution) for conditions where rows in a local table are used to determine rows of a remote table to be deleted (e.g. delete from rmt1 where rmc1 = 12 and rmc2 in (select lc1 from lt1)) where the remote table was linked without a Primary Key primary key specified. Normally the primary key is used, even when not in the original select select, to identify the rows to be deleted. When the primary key is false or not available and SkipDMLPrimaryKey = 1 then the original where clause will be used instead.
This option controls the SQL compiler ability to do some optimizations when it knows it will receive 1 row from a remote table (using the applicable WHERE clause). In some cases the remote is not providing any info about a primary key (or is providing wrong data for the primary keys and unique indices of a table) through ODBC API and the Virtuoso SQL optimizer may get fooled into thinking that a given SQL query over a remote table will return no more than 1 row. To avoid that turn on the above parameter to stop the compiler from doing that optimizations.
This option controls the aggregation point of the VDB connection pools. When it is off (=0, the default) the VDB connections that are no longer needed after transaction end are collected server-side in a per-user connection cache. This (together with the KeepConnectionOnFixedThread INI option) ensures conformance even with older ODBC drivers that require that connections must be used only within the OS thread that initiated them. This however may result in somewhat low reuse rate for the pooled connections. That's why when this option is 1 (On) the connections are stored in a server-side per-DSN connection cache. Keep in mind that this may not work with all the ODBC drivers, but it may provide lower ratio between virtuoso client connections and VDB-to-remote DBMS connections, thus requiring smaller number of active connections on the remotes and faster execution of VDB operations.
String parameter which specifies the path where UCM files are located. If this parameter is not specified, UCM files cannot be loaded.
Every UcmN parameter specifies one UCM file to load. The value of UcmN is a pair of comma delimited strings. The first string is a name of the UCM file to load, (relative to the path specified in UcmPath), the second string is a name of the encoding as it was used by the server. E.g. Ucm1 = java-Cp933-1.3-P.ucm,Cp933 will load the encoding from the file java-Cp933-1.3-P.ucm and associate it with the identifier 'Cp933'. You can register one encoding with more than one name, if they are delimited by '|' (with no white spaces in the string). e.g. Ucm2 = java-Cp949-1.3-P.ucm,Cp949|Korean will load the encoding from the file java-Cp949-1.3-P.ucm and associate it with two identifiers, 'Cp949' and 'Korean'. See UCM Encodings for more details.
Name used to advertise the Virtuoso ODBC service details in ZeroConfig. This is the name that will be shown to clients amongst other ZeroConfig datasources.
An ODBC style connect string to preset the values of the parameters when the ODBC service offered by this server is selected by the Virtuoso ZeroConfig enabled clients.
Name used to advertise the Virtuoso ODBC SSL encrypted service details in ZeroConfig.
An ODBC style connect string to preset the values of the parameters when the ODBC SSL encrypted service offered by this server is selected by the Virtuoso ZeroConfig enabled clients.
The Zero Config section.
The directory containing shared objects/libraries for use as Virtuoso VSEI plugins.
<number> is the module load number, required and starting with 1. <module type> specifies the type of module that is to be loaded, and hence how Virtuoso is to use it. So far "Hosting" and "attach" types exist. <module name> is the file name of the modules shared library or object.
Example:
Load6 = attach, libphp5.so)
Load7 = Hosting, hosting_php.so)
"Attach" is used for now for the php library. It can be used to load other libraries in future too. The reason is to load PHP5 functionality into virtuoso namespace, so when actually is loaded the hosting plugin, it can bind to the already available symbols for php5.
<number> must be ordered from 1 upwards; The <size> is the size of the segment which is equally divided across all stripes comprising the segment.
This section is only effective when Striping = 1 is set in the [Database] section. When striping is enabled the Virtuoso database spans multiple segments where each segment can have multiple stripes.
Striping can be used to distribute the database across multiple locations of a file system for performance. Segmentation can be used for expansion or dealing with file size limitations. To allow for database growth when a file system is near capacity or near the OS file size limitation, new segments can be added on the same or other file systems.
Striping only has a potential performance benefit when striped across multiple devices. Striping on the same file system is needless and unlikely to alter performance, however, multiple segments do provide convenience and flexibility as described above. Striping across partitions on the same device is likely to reduce performance by causing high unnecessary seek times on the physical disk.
Database segments are pre-allocated as defined. This can reduce the potential for file fragmentation which could also provide some performance benefit.
Virtuoso striping alone does not allow for any fault tolerance. This is best handled at the I/O layer or by the operating system. File system RAID with fault-tolerant striping defined should be used to host the Virtuoso files if striping based protection is desired.
The segments are numbered, their segment <number> must be specified in order starting with segment1.
The <size> is the total size of the segment which is that will be divided equally across all stripes comprising the segment. Its specification can be in gigabytes (g), megabytes (m), kilobytes (k) or in database blocks (b) the default.
The segment size must be a multiple of the database page size which is currently 8k. Also, the segment size must be divisible by the number of stripe files contained in the segment.
Segments can be added to a database, however once defined they should never be altered. Databases that were created without striping cannot automatically be restarted with striping. You can convert a non-striping database to striping by dumping the contents of the database to a transaction file and replaying it with striping enabled. An on-line backup made with backup_online () can be restored on a database with a different striping configuration as long as the total number of pages is no less than the number of pages of the backed up database.
Striping can be useful for the temporary objects database if large hash join temporary spaces or such are expected. This is enabled by the Striping setting in the Temp Database section of the ini file. The stripes will be declared in the TempStriping section.
Rebuilding A Database in the Backup section.
Following is the text of the sample virtuoso.ini file that comes with the distribution.
; ; virtuoso.ini ; ; Configuration file for the OpenLink Virtuoso VDBMS Server ; ; ; Database setup ; [Database] DatabaseFile = virtuoso.db TransactionFile = virtuoso.trx ErrorLogFile = virtuoso.log ErrorLogLevel = 7 FileExtend = 200 Striping = 0 ; LogSegments = 1 ; crashdump_start_dp = 0 ; crashdump_end_dp = 0 ; Log1 = log1.trx ; ; Server parameters ; [Parameters] ServerPort = 1111 ServerThreads = 10 CheckpointInterval = 60 CheckpointSync = 2 NumberOfBuffers = 2000 MaxDirtyBuffers = 1200 MaxCheckpointRemap = 2000 PrefixResultNames = 0 CaseMode = 1 ;MinAutoCheckpointSize = 4000000 AutoCheckpointLogSize = 40000000 CheckpointAuditTrail = 1 [HTTPServer] ServerPort = 1122 ServerRoot = ../vsp ServerThreads = 2 MaxKeepAlives = 10 KeepAliveTimeout = 10 MaxCachedProxyConnections = 10 ProxyConnectionCacheTimeout = 15 DavRoot = DAV [Replication] ServerName = log1 Server = 1 QueueMax = 50000 [Client] SQL_QUERY_TIMEOUT = 0 SQL_TXN_TIMEOUT = 0 SQL_PREFETCH_ROWS = 100 SQL_PREFETCH_BYTES = 16000 [AutoRepair] BadParentLinks = 0 BadDTP = 0 ;[Ucms] ;UcmPath = /ucm ;Ucm1 = java-Cp933-1.3-P.ucm,Cp933 ;Ucm2 = java-Cp949-1.3-P.ucm,Cp949|Korean ;Ucm3 = java-ISO2022KR-1.3-P.ucm,ISO2022KR|ISO2022-KR ; ; Striping setup ; ; These parameters have only effect when Striping is set to 1 in the ; [Database] section, in which case the DatabaseFile parameter is ignored. ; ; With striping, the database spans multiple segments ; where each segment can have multiple stripes. ; ; Format of the lines below: ; Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ] ; ; <number> must be ordered from 1 up. ; ; The <size> is the total size of the segment which is equally divided ; across all stripes comprising the segment. Its specification can be in ; gigabytes (g), megabytes (m), kilobytes (k) or in database blocks ; (b, the default) ; ; Note that the segment size must be a multiple of the database page size ; which is currently 8k. Also, the segment size must be divisible by the ; number of stripe files constituting the segment. ; ; The example below creates a 200 meg database striped on two segments ; with two stripes of 50 meg and one of 100 meg. ; ; You can always add more segments to the configuration, but once ; added, do not change the setup. ; [Striping] Segment1 = 100M, db-seg1-1.db, db-seg1-2.db Segment2 = 100M, db-seg2-1.db
When data is inserted into tables, database pages are split and typically pages end up not being fully utilized. If data is inserted in ascending order of key value, which is often the case, space utilization is more efficient. Still, gaps can be left by updates, deletions and page splitting. Virtuoso has an autocompact feature which will take groups of adjacent dirty pages and see if it can fit the same content on a smaller number of pages. It will rewrite the pages to save space before writing these to disk. This operates automatically. Statistics on autocompaction are shown beside the Read ahead status line of the result set of status ('');. The number of pages affected by autocompaction and the number of resulting pages are shown. The numbers are cumulative since the start of the instance.
Autocompact is non-locking and if pages are busy, they will not be touched. Only relatively old dirty pages, about to be written to disk are considered for compaction, not interfering with the hottest part of the working set.
The automatic compaction is not however effective if pages are updated singly, never making stretches of consecutive dirty pages. Therefore a manual compaction function called DB.DBA.VACUUM () is also offered.
The vacuum stored procedure gets an optional table and index name and will read the index from beginning to end. If neither argument is given, all indices of all tables will be compacted. If only the table is given, all indices of this table will be compacted.
If consecutive leaves can be fit on fewer pages than they now occupy, this will rewrite them and free the pages left over. This does however require transient space since the pages are not really replaced until the next checkpoint, hence a vacuum operation can run out of disk. Using the checkpoint statement to force a checkpoint will free the space.
The effects and need for explicitly vacuuming a database can be assessed with the DB.DBA.SYS_INDEX_SPACE_STATS view.
Running:
select * from DB..SYS_INDEX_SPACE_STATS order by ISS_PAGES desc;
produces a result set with the most space consuming index on top. ISS_PAGES is the total count of pages. ISS_ROW_BYTES is the byte count of the rows. If dividing the total count of bytes by the count of pages is much below 8172, (8K - 20), chances are that vacuuming the index may save space. Note that blobs are not affected by vacuuming. If the blobs are small enough to fit on the row as normal strings they are already there. Otherwise they occupy the needed number of pages and cannot be made more compact.
Note that querying the SYS_INDEX_SPACE_STATS view will always read through all the allocated pages of the database and may take a while. The operation is not locking. Only the state as of the last checkpoint will be shown, hence it is a good idea to run the checkpoint statement before querying this view.
Examples:
DB..VACUUM (); -- Compact all tables and indices of the Virtuoso instance
Db..VACUUM ('WS.%'); -- compact all tables of the WS. qualifier
DB..VACUUM ('DB.DBA.RDF_QUAD', 'RDF_QUAD_PGOS'); -- compact the rdf_quad_phgos index of the rdf_quad table.
Virtuoso has an autocompact feature.
This section presents the command line switches of the Virtuoso server executable. Depending on the model and virtual database middleware the server will have different names, all starting with virtuoso-. All these however have the same options for UNIX systems and slightly different for Windows platform.
on Windows platform are available following server command line options:
Usage: virtuoso-odbc-t.exe [-clnCbDARf---dSIMKmrd] [+configfile arg] [+licensefile arg] [+no-checkpoint] [+checkpoint-only] [+backup-dump] [+crash-dump] [+crash-dump-data-ini arg] [+restore-crash-dump] [+foreground] [+pwdold arg] [+pwddba arg] [+pwddav arg] [+debug] [+service arg] [+instance arg] [+mode arg] [+dumpkeys arg] [+manual] [+restore-backup arg] [+debug] +configfile specify an alternate configuration file to use, or a directory where virtuoso.ini can be found +licensefile specify an alternate license file to use, or a directory where virtuoso.ini can be found +no-checkpoint do not checkpoint on startup +checkpoint-only exit as soon as checkpoint on startup is complete +backup-dump dump database into the transaction log, then exit +crash-dump dump inconsistent database into the transaction log, then exit +crash-dump-data-ini specify the DB ini to use for reading the data to dump +restore-crash-dump restore from a crash-dump +foreground run in the foreground +pwdold Old DBA password +pwddba New DBA password +pwddav New DAV password +debug allocate a debugging console +service specify a service action to perform +instance specify a service instance to start/stop/create/delete +mode specify mode options for server startup (onbalr) +dumpkeys specify key id(s) to dump on crash dump (default : all) +manual specify when create a service to make it for manual startup +restore-backup restore from online backup +debug Show additional debugging info The argument to the +service option can be one of the following options start start a service instance stop stop a service instance create create a service instance screate create a service instance without deleting the existing one delete delete a service instance list list all service instances
The below are switches for server for UNIX platforms:
Usage: virtuoso-iodbc-t [-fclnCbDARwMKr---d] [+foreground] [+configfile arg] [+licensefile arg] [+no-checkpoint] [+checkpoint-only] [+backup-dump] [+crash-dump] [+crash-dump-data-ini arg] [+restore-crash-dump] [+wait] [+mode arg] [+dumpkeys arg] [+restore-backup arg] [+pwdold arg] [+pwddba arg] [+pwddav arg] [+debug] +foreground run in the foreground +configfile use alternate configuration file +licensefile use alternate license file +no-checkpoint do not checkpoint on startup +checkpoint-only exit as soon as checkpoint on startup is complete +backup-dump dump database into the transaction log, then exit +crash-dump dump inconsistent database into the transaction log, then exit +crash-dump-data-ini specify the DB ini to use for reading the data to dump +restore-crash-dump restore from a crash-dump +wait wait for background initialization to complete +mode specify mode options for server startup (onbalr) +dumpkeys specify key id(s) to dump on crash dump (default : all) +restore-backup restore from online backup +pwdold Old DBA password +pwddba New DBA password +pwddav New DAV password +debug Show additional debugging info
The +crash-dump option will make use of the segmented log defined in virtuoso.ini for storing the recovery log. See Crash Recovery and virtuoso.ini below for more information. The other options will not use the segmented log.
The +restore-crash-dump option will alter the server startup sequence so that the recovery log produced by +crash-dump will be re-played correctly.
The +mode option can be a combination of the following letters:
On Unix platforms the executable will detach itself from the console and run in the background as a daemon unless the +foreground switch is specified.
For Windows NT and Windows 2000, the Virtuoso server will normally be installed as a Windows service and can be started from the Control Panel or automatically at system startup.
Ordinarily the Windows service will be a system process that runs in the background. If you want the Virtuoso service on Windows to allocate a debugging console the you can use the +debug (-d) switch. This switch is only applicable to starting a service.
Virtuoso on Windows can be run directly from the command line using the +foreground (-f) switch. The server will then start in the foreground of the current "cmd" session. If this switch is not used then the executable on Windows will assume that you are attempting to start a Virtuoso service.
Windows services can be created and removed from the system as required. The default installation under Windows will create a service by the name: OpenLink Virtuoso VDBMS Server, and optionally another service with the name OpenLink Virtuoso VDBMS Server [demo]. The Demo service is a supplied demonstration database that can be installed.
The following options are available to the +service switch for configuring Virtuoso services:
They are used with the +instance <name> where <name> is the instance name to configure a particular instance. All instances are listed in the services applet, with their name in square brackets.
+service list can be used to obtain the list if services that are registered with Windows.
For each service listed you can start, stop, or delete the service.
+service create can be used to create a new service. In this case you also need to specify other start up options that would be associated with the new service entry. If you were using an alternative configuration file this must be specified using +configfile switch.
Make sure the Services Control Panel is closed, before attempting to modify services from the command line, otherwise locking may occur.
The "ZeroConfig" protocol, also known as "Zeroconf" or "Zero Configuration" is a protocol that allows discovery of services on the network that are advertised by their hosts. It also has provisions for automatic discovery of computers and various devices. The main benefit of ZeroConfig is that it does not require DHCP, DNS or directory servers.
ZeroConfig is an open protocol that Apple submitted to the IETF for a standard-creation process.
The Virtuoso server and ODBC driver use the capabilities of ZeroConfig to facilitate DSN (Data Source Name) setup and usage. This is divided in two parts: Server-side and Client-side.
The Virtuoso server (Server-side) is configured via the Virtuoso INI file to advertise its availability on a network with a given name. This allows applications, and in particular the Virtuoso ODBC driver, to receive information about a server, such as its network address, default login, etc, and use it for configuring a data source or directory making a connection.
The Virtuoso ODBC driver (Client side) uses ZeroConfig to locate the desired Virtuoso server during the set-up phase of a data source, and determine available connection options such as:
ZeroConfig provides the client with a service name, which must be bound to the IP address/port of a host of the chosen service during DSN configuration. This is used when existing DSN using a ZeroConfig name is used to connect, it will map name with IP address and port before making a connection.
The Virtuoso server is configured to advertise itself based on the details specified in the [Zero Config] section of the Virtuoso INI file. Below is an example of such:
... [Zero Config] ServerName = Virtuoso Server ServerDSN = UID=demo;PWD= SSLServerName = Virtuoso Server (via SSL) SSLServerDSN = UID=dba;PWD=;ENCRYPT=1 ...
The ServerName and SSLServerName are human readable strings chosen by the administrator to provide clients with a suitable description of the service being provided.
If the Virtuoso does not have the SSL listener enabled then the SSL service will not be advertised automatically. The SSL* keys will simply be ignored and do not need to be removed.
The ServerDSN and SSLServerDSN are default connection strings that can be used by clients to make the advertised connection. You only need to to specify default username and password in these strings. The default database can be specified or left to the setting for the username. You cannot specify the server hostname, IP address or port number, these are supplied by Virtuoso automatically.
ZeroConfig service advertising is multicast, hence it is advertised on all available network interfaces.
Upon DSN set-up the ODBC driver listens for advertising servers, and compiles a roster. This is displayed for the user to choose the desired service to connect to.
If ZeroConfig is used to for data source set-up then the set-up dialog will be initialized based on the details in the connection string configured on the server.
When a DSN is configured based on a ZeroConfig service, the driver will resolve the service name before making the connection to the server. The driver does not store the network address or port number of the Virtuoso server, only the ZeroConfig server name, so if the server's physical address is changed the client DSNs associated with it do not all have to reconfigured; they will resolve to the new address automatically on next use.
The database status report is divided into 6 sections:
This section shows how many connections are open and how many threads the process has and how many are running at the present time. This also displays the number of requests that have been received but are not yet running on any thread.
File size 203161600, 24800 pages, 259 free. 7000 buffers, 6987 used, 3884 dirty 8 wired down, repl age 8251 . Disk Usage: 14246 reads avg 6 msec, 74% read last 10 s, 14457 writes, 4 read ahead, batch = 5. Gate: 2729 2nd in reads, 0 gate write waits, 3372547 in while read 0 busy scrap. Log = wi.log, 9851835 bytes 14950 pages have been changed since last backup (in checkpoint state) Current backup timestamp: 0x0000-0x00-0x00 Last backup date: unknown Clients: 18 connects, max 17 concurrent, 1024 licensed RPC: 54441 calls, 17 pending, 17 max until now, 0 queued, 53988 burst reads (99%), 0 second Checkpoint Remap 7646 pages, 0 mapped back. 0 s atomic time. DB master 24800 total 259 free 7646 remap 3415 mapped back temp 200 total 196 free
The status consists of the following items:
The database file size in bytes or 0 if the database consists of statically allocated files. The total number of 8K database pages follows, then the number of free pages. The number of buffers shown the total count of 8K file cache buffers, followed by the number of used buffers and the number of buffers that are dirty at the time. The wired down count is normally zero but can be transiently other if pages are wired down for processing by threads in the server.
Shows the cumulative total number of reads and writes and the average length of time spent inside the read system call for the database files.
The percentage is the percentage of the real time spent inside read between this status report and the previous status report. This may exceed 100% if several reads are taking place concurrently on different stripes in a multi-file database.
Lists concurrent events. The 2nd in read is the count of concurrent requests for the same page, the gate write waits is the count of times a modify operation had to wait for exclusive access to a page being read by another thread, the in while read is the count of file cache hits that have taken place while a read system call was in progress on another thread.
Thus section shows the count of pages, free pages, checkpoint remap and mapped back for the main database and the space for temporary data such as sort results and hash indices. The page count is the total size, the free count is the count of free pages, the checkpoint remap is the count of pages that occupy two pages in checkpoint space instead of one, the mapped back count is the number of pages that will return to their original place in checkpoint space at the next checkpoint. Understanding these is not necessary.
The Disk Configuration section for a discussion of checkpoint remapping.
The lock section shows various locking statistics accumulated since the server was started. The deadlock count is divided into deadlocks caused by a situation where several transactions read a page and one wants to get write access and all other deadlock situations. The first is called 2r1w deadlock in the report.
The lock section also shows the total number of threads running, i.e. engaged in performing some operation for a SQL or web client. The number of threads waiting is the number of running threads that are presently waiting for a lock. The number of threads in vdb is the number of threads engaged in remote database operations or other 'slow' I/O, such as access to outside HTTP or SOAP services.
All locks currently in effect are listed with the owners (a) and possibly waiting transactions. The transactions are named after their client. A log or replication replay transaction is here named 'INTERNAL'.
Each connected client is listed with the number of bytes sent and received from the client. The transaction status is either PENDING for OK or BLOWN OFF or DELTA ROLLED BACK for a transaction killed by deadlock or timeout. The locks owned by the transaction are listed following the status. IE means exclusive and IS shared lock.
This section shows the server in question and a list of replication accounts either provided or received by this server. Accounts where the server name (left column) is the same as the server name are those provided by this.
The columns are server name, account name, last transaction number and status. The status is OFF for a local account or a replicated account where the remote is not available. It is SYNCING if a resync is in progress, IN SYNC if the account is up to date or REMOTE DISCONNECTED if there was a connection to a remote party which subsequently disconnected.
This part of the report summarizes the database's access statistics. The output is a table with a row for each index in the database. Each row is composed of the following columns:
Table The name of the table Index The name of the index. Same as the table for primary key. Touches The number of touches since startup. Each time the database engine looks at an entry of the index is counted as a touch. Not all touched entries are selected. For instance if the engine scans a table with non-indexed selection criteria it will touch each row but might select none. Reads The number of disk reads caused by reading this index. %Miss The percentage of touches that required a read. This can be over 100% since getting one entry may required more than one read if the top levels of the index are not in memory. Locks The number of times a lock is set on an entry of the index. Waits The number of times the engine has to wait for another transaction to finish in order to set a lock on this index. %W The percentage of waits of all locks set.
In interactive SQL
SQL> status();
Will print out the report.
The Virtuoso Service name can be altered using the key Win32ServiceName in the Parameters section. The default name is 'OpenLink Virtuoso VDBMS Server'
To change the name of services:
Services with old names must be deleted before creating service with the new name, i.e. with the Win32ServiceName setting set to the current name of the service.
The name displayed in the ODBC Administrator, Setup and Configuration dialogs is taken from the driver section in the ODBCINST. This can be directly edited in the registry using the regedt32 Windows utility, or a registry import file can be created which can be applied by simply double-clicking the .reg file. Always exercise extreme caution when making changes to the registry.
Virtuoso has the ability to encrypt it's CLI network connections using SSL. The server listens on a separate port for SSL CLI connections and handles them just as the normal CLI connections, vut now providing transport level security.
Server side secure connections utilizes three parameters in the [Parameters] section of the virtuoso.ini:
These parameters should be all set in order to enable the SSL CLI server.
If SSLServerPort is not specified, then the Virtuoso server ignores the other two and does not listen for SSL CLI connections.
If a non-SSL connection is attempted to the SSL server port, the server rejects the connection. If an SSL connection is attempted against the non-SSL port the server rejects the connection.
The client does not require any SSL-specific files (like Certificates or Private keys) in the SSL connection process.
There is an custom ODBC connect option SQL_ENCRYPT_CONNECTION (=5004) supported by the Virtuoso CLI. It should be set before issuing the SQLConnect call. Values are 'NULL' (no encryption - default), '1' (encryption with no server X509 certificate checking and no X509 certificate sent to the server) and a valid file path to a PKCS#12 certificate file (protected with the same password as the one used to log in. Note that with the iODBC/ODBC clients this connect option is not applicable since the driver managers don't cache or pass through the custom ConnectOptions set before connecting to the data source. The ISQL has an additional option (-E) to do encrypted connects using the encryption option '1') and -X <file> to set the above option to the file supplied.
The drivers support an additional DSN attribute:
ENCRYPT=<string>
If this attribute is not specified then it defaults to "No".
It has the same meaning as the SQL_ENCRYPT_CONNECTION options (see above). If this is not specified then it defaults to NULL.
The corresponding iODBC odbc.ini & ODBC Registry DSN attribute name is:
"Encrypt"= <string>
The Windows Connect & Setup dialogs have an additional wizard page to configure encryption.
Virtuoso supports X509 certificate validation: server side for both ODBC and HTTPS connections, and client side for the ODBC connections.
To enable this option there are three new INI file parameters added the HTTPServer section for the HTTPS, and the Parameters section for ODBC):
In order for verification of the server certificate to take place a PKCS#12 file should be supplied to the ODBC client. It will use the CA list in this PKCS#12 to verify the server certificate. It will set the verification depth to -1 (unlimited) while performing such a check.
If the server certificate is not verified correctly it will refuse to connect to the server. When the ENCRYPT parameter is set to "1" (do SSL without X509 validation) the client will return a SQL_SUCCESS_WITH_INFO in SQLConnect/SQLDriverConnect with the Server's certificate subject and the verification result as the server will always send it's X509 certificate to the client as a part of the SSL connect handshake.
If the PKCS#12 file is supplied the ODBC client will try to open it using the login password. In order for the file to be successfully opened it should be encrypted with the same password used for logging in.
Normally when exporting a PKCS#12 file from other programs it will contain only the CAs of the Certificate validation chain. This means that client and server certificates should have common CA in their certificate chains in order to be used for ODBC X509 validation. The client certificate from the PKCS#12 file will not take place in the server certificate validation process.
Access Control Lists (ACL) are used to restrict file system access.
These lists are maintained in the Virtuoso INI file under the Parameters section with entries such as:
DirsAllowed = <path> [, <path>] DirsDenied = <path> [, <path>] <path> := <absolute_path> or <relative_path>
A relative path is relative to the servers current working directory.
The Virtuoso ISQL utility can be used to check the Server DirsAllowed params as follows:
SQL> select server_root (), virtuoso_ini_path ();
The above should show in the result the server working directory and INI file name.
Also you can check the relevant INI setting by running following statement via ISQL command line utility:
SQL> select cfg_item_value (virtuoso_ini_path (), 'Parameters', 'DirsAllowed');
ACL's work in the following way:
The following functions are restricted by file Access Control Lists (ACL) in the virtuoso.ini file:
the cfg_write function has restrictions against changing file access control lists in ini file
The Virtuoso Server supports linking in of tables, views, procedures and other components from a remote ODBC data-source, enabling them to be queried as native objects in Virtuoso; this process is called "attaching" or "linking". The easiest way to link to an external table is to use the Linking Remote Tables Wizard, part of the Visual Server Administration Interface. Alternatively you can attach these objects programmatically, as this section explains; finally you can attach tables manually - see Manually Setting Up A Remote Data Source which is useful for connections to less-capable ODBC data-sources.
ATTACH TABLE <table> [PRIMARY KEY '(' <column> [, ...] ')'] [AS <local_name>] FROM <dsn> [USER <uid> PASSWORD <pwd>] [ON SELECT] [REMOTE AS <literal_table_name>]
table: |
Adequately qualified table name of the form: identifier | identifier.identifier | identifier.identifier.identifier | identifier..identifier |
column: |
column to assume primary key |
local_name: |
fully qualified table name specifying local reference. |
dsn: |
scalar_exp |
user: |
scalar_exp |
password: |
scalar_exp |
literal_table_name: |
scalar_exp |
This SQL statement defines a remote data source, copies the schema information of a given table to the local database and defines the table as a remote table residing on the data source in question.
The table is a designation of the table's name on the remote data source dsn. It may consist of an optional qualifier, optional owner and table names, separated by dots. This must identify exactly one table on the remote dsn. The optional local_name is an optionally qualified table name which will identify the table on the local database. If qualifier or owner are omitted, these default to the current qualifier 'dbname()' and the logged in user, as with CREATE TABLE. If the local_name is not given it defaults to the <current qualifier>.<dsn>.<table name on dsn>. The <dsn> will be the dsn with all alphabetic characters in upper case and all non-alphanumeric characters replaced by underscores. The <table name on dsn> will be the exact name as it is on the remote dsn, case unmodified.
The PRIMARY KEY option is only required for attaching views or tables where the primary key on the remote table cannot be ascertained directly from the remote data source.
If a dsn is not previously defined with vd_remote_data_source or ATTACH TABLE, the USER and PASSWORD clauses have to be given.
The REMOTE AS option allows you to provide a string literal for referencing the remote table. This is useful when linking tables from sources that do not support three-part qualification correctly.
A view can be attached as a table if a set of uniquely identifying columns is supplied.
This is done with the PRIMARY KEY option to ATTACH TABLE as follows:
attach table T1_VIEW primary key (ROW_NO) from 'somedsn';
Views cannot be attached unless the PRIMARY KEY options is used.
ATTACH (PROCEDURE|FUNCTION) <proc_name> ([<parameter1>[,<parameter2>[...]]]) [ RETURNS <rettype> ] [AS <local_name>] FROM <dsn>
dsn: |
scalar_exp |
proc_name: |
identifier | identifier.identifier | identifier.identifier.identifier | identifier..identifier |
parameter1..parameterN: |
parameters declaration (as in CREATE PROCEDURE) |
local_name: |
table |
The ATTACH PROCEDURE statement allows you to associate stored procedures from remote datasources with Virtuoso so they can be used as if they were defined directly within Virtuoso. Much like the ATTACH TABLE statement, this SQL statement creates a local alias for a procedure on a given remote data source so it can be considered locally defined. When this alias is called called the procedure at the remote data source will actually be called.
Procedure generated result sets are not supported by the ATTACH PROCEDURE statement. The only portable way to return values from a remote procedure is to use INOUT or OUT parameters. Remote procedure result sets can be used by combination of rexecute() and Virtuoso PL, but this is left for the user to implement as required.
The ATTACH PROCEDURE statement is not able to define new connections to remote data sources, the connection should be defined prior using either vd_remote_data_source or by attaching a table or view using the ATTACH TABLE statement with USER/PASSWORD supplied.
Note that when generating pass-through statements to a given remote, any procedure call for an attached procedure is passed through if the current DSN is the same as the remote procedure's DSN.
The proc_name is the designation of the procedure's name on the remote data source, DSN. The remote procedure name supplied should always be fully qualified to avoid ambiguity, it may consist of an optional qualifier/catalog, optional owner and finally procedure name, separated by dots. This must identify exactly one procedure on the remote data source.
The optional local_name is an optionally qualified procedure name which will identify the procedure on the local Virtuoso database. If the local_name is not given it defaults to the <current qualifier>.<dsn>.<proc name on dsn>. The <dsn> will be the data source name in upper case and with all non-alphanumeric characters replaced by underscores. The <proc name on dsn> will be the exact name as it is on the remote dsn, case unmodified.
If a dsn is not previously defined with vd_remote_data_source or ATTACH TABLE, the ATTACH PROCEDURE will fail.
On remote Virtuoso (DSN name : remote_virt):
CREATE PROCEDURE RPROC (IN PARAM VARCHAR) returns VARCHAR { return (ucase (PARAM)); }
On the local virtuoso (DSN name : local_virt) :
vd_remote_data_source ('remote_virt', '', 'demo', 'demopwd'); ATTACH PROCEDURE RPROC (IN PARAM VARCHAR) RETURNS VARCHAR from 'remote_virt';
will result in creation of an procedure alias for RPROC in local_virt named DB.REMOTE_VIRT.RPROC
Calling it from the local_virt (using ISQL)
select REMOTE_VIRT.RPROC ('MiXeD CaSe') as rproc_result; rproc_result --------------- MIXED CASE 1 rows
The Virtuoso Visual Server Administration Interface provides a graphical user interface for linking remote stored procedures.
If a statement is passed through to a remote data source, the types returned by SQLDescribeCol are taken directly from the remote prepare and converted to the closest Virtuoso supported type.
If a statement involves both local and remote resources all types are taken from the Virtuoso copy of the data dictionary.
In executing remote selects Virtuoso binds output columns according to the type and precision given by SQLDescribeCol after preparing the remote statement.
When a table is attached from a remote data source the catalog is read and the equivalent entries are created in Virtuoso. Since the types present on different DBMS's vary, the following logic is used to map ODBC types to Virtuoso types.
SQL Type | Mapped Type |
---|---|
SQL_CHAR | varchar (precision) |
SQL_VARCHAR | varchar (precision) |
SQL_BIT | smallint |
SQL_TINYINT | smallint |
SQL_SMALLINT | smallint |
SQL_INTEGER | integer |
SQL_BIGINT | decimal (20) |
SQL_DECIMAL | |
SQL_NUMERIC |
smallint if precision < 5 and scale = 0 integer if precision < 10 and scale = 0 double precision if precision < 16 decimal (precision, scale) otherwise |
SQL_REAL | real |
SQL_FLOAT | double precision |
SQL_DOUBLE | double precision |
SQL_BINARY | varbinary (precision) |
SQL_VARBINARY | varbinary (precision) |
SQL_LONGVARCHAR | long varchar |
SQL_LONGVARBINARY | long varbinary |
SQL_DATE | date |
SQL_TIME | time |
SQL_TIMESTAMP | datetime |
The general case of decimal and numeric both revert to the Virtuoso decimal type, which is a variable precision decimal floating point.
One transaction on the Virtuoso VDBMS server may contain operations on multiple remote data sources. As a general rule remote connections are in manual commit mode and Virtuoso either commits or rolls back the transaction on each of the remote connections as the main transaction terminates.
ODBC does not support two phase commit. Therefore a transaction that succeeds on one remote party may fail on another.
A transaction involving local tables and tables on one remote data source will always complete correctly since the remote is committed before the local and the local is rolled back if the remote commit fails.
Note that even though the client to Virtuoso connection may be in autocommit mode the continuing connections will typically not be autocommitting.
A remote connection is in autocommit mode if the Virtuoso connection is and the statement is passed through unmodified. In all other cases remote connections are in manual commit mode.
Virtuoso supports 2PC - Two Phase Commit. See the Distributed Transaction & Two Phase Commit section for more information.
Different DBMS's support slightly different sets of SQL built-in functions with slightly differing names. For example, what is called substring on Virtuoso is called substr on Informix. Virtuoso allows declaring equivalences between local user-defined or built-in functions and user defined or built-in functions on remote servers. Knowing that a function exists on a remote database allows passing processing closer to the data, resulting in evident gains in performance.
To declare that substring is called substr on DSN inf10, you can execute:
db..vd_pass_through_function ('inf10', 'substring', 'substr');
The first argument is the name of the remote database, as used with attach table and related statements. If user defined functions with qualified names are involved, the names should be qualified in the vd_pass_through_function call also. If many qualified or unqualified forms of the name are in use, one should declare the mapping for them all.
To verify whether this declaration takes effect, one can use explain to see the query execution plan, for example:
explain ('select substring (str, 1, 2) from inf10.sample_table');
The declarations are persistent and can be dropped by using a last argument of NULL for a given function. The declarations are kept at the level of a DSN and not at the level of the type of DBMS because different instances can have different user defined functions defined.
If a query can be executed in its entirety on a single remote database, then optimizing this query is exclusively the business of the remote database, as it gets the whole query text. Virtuoso rewrites some things and suggests a join order but these are not binding on the remote database.
If a query involves tables from multiple remote databases or a a mix of local and remote tables, knowing basic SQL statistics on the tables is essential for producing a meaningful query plan. Virtuoso has information on indices existing on remote tables and if the remote table is attached from a known type of DBMS, Virtuoso may read the DBMS specific statistics at attach time.
Note that the statistics of the remote database should be up to date before attaching.
The function sys_db_stat can be used for forcing a refresh of Virtuoso's statistics on remote tables.
sys_db_stat (5, 0)
will go through all tables, local and remote. For local tables, it will update statistics with a 5 percent sampling rate and for remote tables it will refresh the statistics if the type of the host DBMS is among the supported ones. If the remote DBMS is of an unknown type, Virtuoso will take the count of the remote table and select the 1000 first rows to get a sample of data lengths and column cardinalities. This is not very precise but will be better than nothing.
In order to force a full read of a remote table for statistics gathering, one can use
db..sys_stat_analyze ('fully qualified table name', 0, 0);
The table name is case sensitive, with all qualifiers, as it appears in SYS_KEYS and other system tables. This will read the whole table.
Statistics on local as well as remote tables are kept in SYS_COL_STAT. One may look at this table to see the effects of remote statistics collection. In special cases, if a special effect is desired or the information is not otherwise available, as in the case of a very large table on an unsupported type of server, it is possible to manually update the contents of the table. Shutting down and restarting Virtuoso will force a reload of the statistics information.
Presently Oracle, Informix and Virtuoso are supported for direct access to the remote database's statistics system tables. It is possible to define hook functions for accessing this same information from any other type of DBMS. Please contact support for instructions on this.
When a query contains mixes of tables from different sources, the compiler must decide on an efficient execution plan that minimizes the number of round trips to remote servers and evaluates query conditions close to the data when possible. Additionally, any normal query optimization considerations such as choice of join order and join type apply. See the section on SQL optimization and optimizer hints for more on this. Additionally, the SQL optimizer uses round trip time statistics for the servers involved in the query.
In the following examples, we use the tables r1..t1, r2..t1 and t1, of which r1..t1 is on a server close by, r2..t1 on a server farther away and t1 on the local server. The column row_no is a unique key and the string1 column is in indexed with 300 distinct values, the column fs1 has 3 distinct values. The tables all have 100000 rows. A round trip to r1 takes 10 ms and a round trip to r2 takes 100 ms.
Consider
select * from r1..t1 a, t1 b where a.row_no = b.row_no and a.fs1 = 'value1';
The compiler notices that 33333 rows will be selected from r1..t1 based on fs1. It will decide to read these into a hash table, causing one linear scan of r1..t1 with relatively few round trips. Then it will read t1 locally and select the rows for which there is a matching entry in the hash. This is slightly faster than doing 33333 random lookups of t1. If fewer rows were selected from r1..t1, the compiler would do a loop join with the local t1 as the inner loop.
The absolute worst plan would be a loop join with t1 as the outer loop, with 100000 round trips to r1.
Now, if many tables are accessed from the same data source, the compiler tries to bundle these together into one statement. Thus, for:
select * from r1..t1 a, r1..t1 b, t1 c where c.string1 = '111' and b.row_no = c.row_no and a.row_no = b.row_no + 1;
The compiler will probably do the outer loop for t1, which is expected to select 100000/300 rows. Then it will do a round trip to r1 with the statement.
select * from t1 a, t1 b where a.row_no = b.row_no + 1 and a.row_no = ?.
This is likely better than doing the remote part as an outer loop, bringing all the approx 100000 results in. 333 round trips selecting 1 row is better than 100000 rows transferred. If the data source were further away, this could be otherwise, hence the importance of the round trip time statistic.
In distributed queries, the compiler will honor the option (order) and the join types e.hg. table option *(hash) insofar the tables are local.
Thus, if we wrote
select * from r1..t1 a, t1 b, r1..t1 c where c.string1 = '111' and b.row_no = c.row_no and a.row_no = b.row_no + 1 option (order);
the compiler could not merge the two tablesfrom r1 into a single query because the order were given and there is an intervening table not located on r1.
ODBC and other data access API's usually offer a mechanism for executing a single parametrized statement multiple times with a single client-server round trip. This is usually called support of array parameters.
Virtuoso can make use of array parameter support in ODBC drivers when available. Consider the statement:
insert into r1..t1 select * from t1;
Without array parameters, this would make a round trip to r1 for each row inn t1. With array parameters enabled, with a batch size of 100, this would make only 1000 round trips for 100000 rows, resulting in dramatic increase in performance. Typically, if the remote server is on the same machine, array parameters make such batch operations about 3x faster. If the remote is farther away, the gain is greater.
Array parameters are used if the remote database and its ODBC driver support them. The feature is globally disabled in the default configuration because some ODBC drivers falsely claim to support array parameters. To enable this feature, the the ArrayOptimization entry in the [VDB] section of the ini file to 1. To set the batch size, use the NumArrayParameters setting. 100 is a reasonable value.
Some ODBC drivers also support array parameters for select statements. To enable using this, you can set the ArrayOptimization setting to 2. This may however not work with some drivers even if DML (insert/update/delete) statements do work with array parameters.
A transaction timestamp is not the same across the transaction if the transaction has branches in different databases.
The data type and precision of a time stamp will also vary between different types of databases.
Hence timestamp columns coming from tables on different servers are not comparable for equality.
In inserts and updates of remote tables timestamps are assigned by the database where the table in question is physically located.
Identity or autoincrement columns are likewise handled by the database holding the remote table.
Note that MS SQL Server and Virtuoso describe a timestamp column as a binary column in ODBC catalog and meta data calls. Thus remote SQL Server or Virtuoso timestamps will not appear as timestamps at all.
In the case of a Virtuoso remote database the binary timestamp can be cast into a DATETIME data type and it will appear as a meaningful datetime.
These procedures allow you to manually manage remote data sources and their tables.
Functions capable of returning a result-set make use of the results_set parameter. To prevent them from returning a result-set, the results_set parameter should be set to 'null'. If Virtuoso finds an awaiting parameter to contain results_set it will fetch the result set regardless of cursor_handle parameter.
Unless explicitly granted, only the DBA group is permitted to use the rexecute() to maintain security. Caution is required here since any user granted use of rexecute() has full control of the remote data source set-up by the DBA, however limited to the overall abilities of the remote user on the remote data source. Users can be granted and denied access to this function using the following commands:
GRANT REXECUTE ON '<attached_dsn_name>' TO <user_name> REVOKE REXECUTE ON '<attached_dsn_name>' FROM <user_name>
The following remote catalogue functions help you to obtain information about the remote datasources that you are using. These could be especially useful in Virtuoso PL later on if you are not able to know everything about the remote tables ahead of time for the ATTACH TABLE statement
Defining a remote table involves declaring the table as a local table and then defining the data source if not already defined and associating the new table with the remote data source.
The data source on which a table resides is declared at the table level. This has no connection to the table's qualifier.
Assume a remote ODBC data source named test containing a table xyz declared as follows:
CREATE TABLE XYZ ( A INTEGER, B INTEGER, C INTEGER, PRIMARY KEY (A));
To defined this as a remote table on the data source Virtuoso, first define the table locally, using the above CREATE TABLE statement above.
Then define the data source:
DB..vd_remote_data_source ('test', '', 'sa','');
And the table:
DB..vd_remote_table ('test', 'DB.DBA.XYZ', 'master.dbo.XYZ');
This assumes that the remote data source has a login 'sa' with an empty password and no special connection string options. The table names in vd_remote_table have to be fully qualified. We here assume that the Virtuoso table was created by DBA in under the default qualifier DB and the remote XYZ was created by dbo in master.
The vd_remote_table declaration does not affect statements or procedures compiled prior to the declaration.
Additional indices of remote tables may optionally be defined. They do not affect the operation of the SQL compiler. The remote data source makes the choice of index based on the order by clause in the statement passed through.
Never attempt to attach a local table as a remote. The server will hang if it tries to make a remote commit on itself.
If the schema of the remote table is changed it will need to be re-attached to Virtuoso.
The Virtuoso server treats dots (.) in the double-quotes escaped names as name element separators. For example : the table name "a.b.c" is treated as "a"."b"."c" . Because of this remote tables with dots in their table name (like tables from MS Text driver) require the dot inside the table name to be replaced with the VDB "non-delimiting-dot" (\x0A) and the vd_attach_table (in dsn varchar, in remote_name varchar, in local_name varchar, in uid varchar, in pwd varchar) to be used instead of ATTACH TABLE statement.
The statement ATTACH TABLE "datafile.txt" as 'test' from 'text' user 'a' password 'b' should become :
vd_attach_table ('text', 'datafile\x0Atxt', 'test', 'a', 'b');
When Virtuoso interacts with a table or view attached from a remote data source, it must be able to uniquely identify each row of the query. At the attach time Virtuoso will query remote data source for the tables primary keys and indices. These will be used to construct a copy of the table definition in Virtuoso which is then used in reference to the remote data source. At query time this information is used as much as possible. This information may need to be supplemented by calls to SQLStatistics() for further indicies or primary key information, as a last resort Virtuoso will use SQLColAttribute() to determine which columns are SQL_DESC_SEARCHABLE.
The Virtuoso User Model is designed to support:
There is a set of functions for administering the users and groups (roles) of a Virtuoso database. All the user administration functions are restricted to members of the dba group only.
The terms 'role' and 'group' are identical in their usage in this document. The terms security object or grantee refer collectively to users and groups.
User manipulation functions do not generally return values. In case of error an error condition will be signaled.
Users and roles are in the same namespace and are identified by name. One name will not both designate a user and a group. A user or group may be SQL enabled. A name that identifies a role (group) does not have a password and is not a valid login name.
A user may belong to multiple roles. A role may inherit multiple other roles. One role may be inherited or directly granted to a security object multiple times. The order of granting is not relevant. The effective privileges of a user are those granted to public, those granted to the user plus those granted to a direct or indirect role of the user. The relationship between users and roles is many to many. A role can be granted multiple other roles as well as direct permissions. Cycles in granting roles are not allowed, an error will be signaled if an operation would result in a cycle in the role inheritance graph.
When SQL procedures or queries are executing, the effective privileges are those granted to the owner of the procedure or view. A top level dynamic SQL statement executes with the effective privileges of the logged in user. If a SQL statement executes as a result of an HTTP request, the applicable virtual directory specifies on behalf of which SQL account the SQL is executed. A role cannot be the owner of procedures, views or other executable SQL entities.
The following functions allow for creation and deletion of security objects and roles, and for assigning or removing roles from security objects:
The security objects and roles data are contained in the system tables described in the User System Tables Section of the Appendix
Function for checking a given password on SQL or DAV login. See below.
Application specific data for the Password Mode hook.
Default qualifier for SQL session.
If set SQL login is granted.
If set the user account can be user for web authentication.
If set the user account is locked and cannot be used to login as SQL or Web user (depends of SQL_ENABLE and DAV_ENABLE flags). If the account in question is SQL enabled the DBA group can switch the execution identity to it (see set_user_id () function). This is useful when we need an account to execute Web pages (VSP/VSPX) with some execution permissions but we do not want to allow it to login via SQL/ODBC.
This is the primary group of the user. This is no different from other group memberships.
Function that will retrieve the password. If not defined the password is assumed to be in the SYS_USERS table. This allows for custom encrypted storage of passwords etc. This is simpler to use than the check hook. Note that for security configurations where the server never does know the passwords of user accounts, no digest based authentication schemes can be used, including the HTTP digest authentication, since the digests cannot be computed and checked without knowing the password. Possible users of this feature are DBEV_LOGIN or HTTP login hooks.
informative: e-mail of that user.
informative: full name of the user.
WebDAV home directory of the account, it is meaningful only if the account is web enabled.
WebDAV default permissions for new WebDAV objects created by the user. This is only meaningful when web access is enabled.
The functions for setting/getting these options will accept any other named values, the above list only being those reserved for Virtuoso so far.
DB.DBA.USER_FIND (in name varchar);
This is a user-defined PL function hook which, if it exists, will be executed before doing the SQL/ODBC login. In this hook the user can find a user account from some other server and register it in the local database. Or, this can be used to perform some pre-login actions. It is similar to the DBEV_LOGIN, but it does not change any account validation rule, it is purely for pre-processing.
The Database Event Hooks chapter.
create procedure DB.DBA.LDAP_SEARCH (inout user_name varchar, in digest varchar) { whenever sqlstate '28000' goto ldap_validation_failure; if (lcase(user_name) <> 'dba')) { ldap_search('foo.bar.com', 0, 'o=organization', '(cn=a*)', sprintf('uid=%s,ou=users,o=organization', user_name), pwd_magic_calc(user_name,digest,1)); user_name := 'dba'; return 1; -- force validation as dba } else { -- bypassing ldap authentication for dba, let validation occur normally return -1; } ldap_validation_failure: return -1; -- try to validate normally }
create procedure DB.DBA.DBEV_LOGIN (inout user_name varchar, in digest varchar, in session_random varchar) { declare get_pwd varchar; get_pwd := user_get_option (user_name, 'PASSWORD_MODE'); if (get_pwd is not null) { declare rc integer; rc := call (get_pwd) (user_name, digest); return rc; } return -1; };
user_create ('test_ldap', 'secret', vector ('PASSWORD_MODE', 'DB.DBA.LDAP_SEARCH'));
create table MY_DBA_USERS (M_NAME varchar primary key, M_PASSWORD varchar);
create procedure DB.DBA.USER_FIND (in name varchar) { -- do nothing for existing users if (exists (select 1 from SYS_USERS where U_NAME = name)) return; -- if there is in custom table if (exists (select 1 from MY_DBA_USERS where M_NAME = name)) { declare pwd varchar; -- get password select M_PASSWORD into pwd from from MY_DBA_USERS where M_NAME = name; -- create a new SQL user based on external data USER_CREATE (name, pwd, NULL); } };
The terms user group and role are used interchangeably. Roles can be nested. There is a many to many relationship between users and roles. There is likewise a similar, acyclic many to many relationship between roles. Each role has a component role list of its granted (parent) roles, recursively, no cycles allowed.
All role grants are listed in the roles system table whether they be explicitly granted or only as a result of granting a group with groups granted to it. The role grant graph has an explicit edge for each role membership, direct or otherwise. The GI_DIRECT flag is true if the grant is direct. Only direct role grants can be revoked.
The role system table description can be found in the appendix under System Tables.
The following SQL statements deal with roles. To create a new role (group) object the following statement can be used:
CREATE ROLE <NAME>
The <NAME> is a name of role to be created. It must be unique in space of all security objects.
SQL> create role admins;
Use the following statement to remove an existing role from the security schema.
DROP ROLE <NAME>
SQL> drop role admins;
The GRANT and REVOKE statements are used for controlling role membership as follows: To assign a new group, or list of groups (<ROLE>,...) to user <USER> use:
GRANT <ROLE> [, <ROLE>] TO <USER> [WITH ADMIN OPTION];
If the admin option is specified, the grantee can grant this same privilege further to other grantees.
Roles can be revoked using:
REVOKE <ROLE> [, <ROLE>] FROM <USER>;
SQL> grant admins, users to demo;
SQL> revoke admins from demo;
Only the dba group accounts may administer roles.
The dba group is not physically a role. When an empty database is created, it will have the dba account with full privileges. To grant these same full privileges to another user, the dba uses the grant all privileges to <grantee>. statement. This will give the grantee full dba privileges, or in other words, add to the dba group. This may be reversed with the revoke all privileges from <grantee> statement.
The GRANT statement accepts any valid SQL security object in the TO clause. One cannot log in or perform any operations as a role. Roles are exclusively shorthand for specific sets of permissions which are changed together and are needed for multiple users.
VAD provides a package distribution framework for installation, management, dependency checking and un-installation of Virtuoso applications. A VAD package contains all required Virtuoso components, which would constitute an application or hosted solution, within a single distributable file. A VAD package cannot contain any system parts independent of Virtuoso thus excluding operating system executables, shared objects, installers or settings.
Virtuoso and VAD provide the following abilities:
The following is what the dba needs to know about VAD packages.
A VAD package is installed from a file with the db..vad_install SQL function. The first argument is the file path, which must be in a location that the server process can open, i.e. it is in the DirsAllowed list in the virtuoso.ini file. The second argument is 0, meaning that we are installing from a file.
SQL> vad_install ('conductor_dav.vad', 0);
is an example. If the package installation fails, the server exits and will have to be restarted. No effects of a failed installation will remain in the database after restart. Contact the supplier of the VAD package for further instructions.
To know what is installed, do:
SQL> vad_list_packages ();
VAD package installations are not recorded in the transaction log. Thus, if there is a backup followed by archived transaction logs produced if CheckpointAuditTrail is on in virtuoso.ini, the VAD install must be performed before replaying any logs that were made after the VAD installation. The package installation must be just in the right place in the replay sequence. In practice it is simplest to make an incremental backup after installing and packages, see backup_online () or the section on backing up.
For any further information, including how to make VAD packages, see the rest of this chapter.
A VAD package has no developer tie-ins; it is built in a development environment from source code that can be managed and versioned in the developers system of preference.
The VAD package is described by an XML structure called the 'VAD Sticker'. The VAD sticker describes items to deploy, procedures to execute at install and uninstall time and all conditions to be checked to ensure correct installation. The VAD Sticker consists of the following:
All required packages should be listed in the VAD sticker. Known conflict may be listed in either of the conflicting VAD packages stickers, hence VAD stickers of all installed packages should be checked.
Later versions of a package may be installed replacing earlier versions of the same package. This however can be prohibited by listing either version (or limit) as a known conflict in either VAD package sticker in the usual way. Furthermore, it is possible to prevent re-installation of a package by stating that it conflicts with itself. This provides some security against exploits involving attempts to upgrade, downgrade or re-install a package, in the hope that the administrator may corrupt the existing installation by installing new packages and working through installing their dependencies.
Packages may differ in language and encoding of documentation and resource files, even though the version number remains the same. If a package is sensitive to internationalization issues, the developer should either assign different names to various localizations of the package, or divide it into kernel package for any language-independent parts and set of language-specific packages, with some dependency between them.
During creation of a VAD package, the "location" mentioned above may be name of a file in file system or URI or DAV path. Upon package-time, URIs will be resolved and resources under them will be copied into the package. The resulting sticker will thus contain the location of resource within the package, the resource itself, and the target location.
All SQL files have a specific order of loading. Tables, views etc. must be defined before being referenced.
The VAD specification explicitly does not define the following:
There are no specific restrictions for the schema or Virtuoso/PL code of the package. The VAD system does not make assumptions on the method of software development.
Version numbers used in the sticker have nothing common with tag labels in a developers versioning system. Procedures edited directly within the database using a web interface or CASE tools should be exported to a file for inclusion in a VAD package. If the application developer uses some script to export such code, this script is not usually part of sticker or the resulting package.
VAD provides no methods for downloading dependent packages, or check for package updates etc.
There can be no guarantee that pre- or post-installation checks will provide valid results if more than one VAD is being processed at the same time. VAD does however guarantee that a package installation will be either entirely successful or entirely rolled back.
Unlike Virtuoso-based packages, these components are usually operating system specific, they may require some complex tuning, and their usage from within Virtuoso applications may even require changes in virtuoso.ini configuration file. VAD packages may contain test calls in pre-installation SQL procedures to check that required external executables are available and provide the functionality required.
Some installations may require several days to complete migration/conversion of stored data. Whilst it may be possible to provide a restricted service during such time, VAD contains no tools to simplify such a process, this is left to the administrator or developer. VAD completes its work right after the execution of the post-installation code.
VAD has no standardized metadata regarding replication issues, hence package-specific code may be required. Similarly, if a cluster uses "round-robin" or a "director" loading management system and the server should be stopped for VAD installation, the administrator should explicitly inform the cluster manager about this event.
Since VAD packages are run by an administrator as the database DBA user, care must be taken to ensure the package comes from a safe source. Any new package installed may violate the security regulations of the target database and may even inflict damage to files under the web-root of the Virtuoso Server or in directories specified in the “DirsAllowed” parameter of the virtuoso.ini. If the virtuoso.ini parameter “AllowOsCalls” is enabled then the installation procedures of the package may call operating executables. It is the responsibility of the database administrator to control this via the “AllowOsCalls”, "SafeExecutables" and "DbaExecutables" parameters of the virtuoso.ini.
VAD packages do not offer any automatic protection against unauthorized modifications. Although ever VAD package contains a checksum, its purpose is to guard against data transfer errors, it may not be sufficient to detect unwanted modification.
Initially, the VAD sticker and resources may reside in the file system, DAV directory and or other locations available through the DB.DBA.HTTP_URI_GET() function.
The VAD creation operation parses the VAD stickerÂ’s XML description and constructs the VAD file.
DB.DBA.VAD_PACK (in sticker_uri varchar, in base_uri_of_resources varchar, in package_uri varchar); returns varchar
This function reads the VAD sticker identified by the sticker_uri which contains the vad:package root element. Then the resources identified in the sticker are retrieved. All resource URIs are interpreted in the context of the base_uri_of_resources and are parsed and checked to be syntactically correct. Resources are appended to generated package that will be stored at the package_uri. Vad_pack() returns a human readable log of error and warning messages, it will signal errors if any resource or database objects are unavailable at build time.
By convention, VAD package files have the extension '.vad'.
An optional VAD package named VADutils provides various tools for capturing changes made in the database after some point in time. The result of a capture consists of:
The capture results may be useful for the following purposes:
These mechanisms provide good support for centralized development and custom deployment methodology. If a site is localized to contain local links, graphics, custom layout and such, then VAD capabilities offer help to the developer to define the specific overlay of customizations over another VAD package. When the underlying VAD package is updated the local customizations will be overwritten. Being saved in a VAD package, customizations can be reapplied over the updated base package.
VAD package installation, upgrade and uninstallation requires a temporary break of service. The package checks may be performed on the fly if it can be guaranteed that the resources being inspected will not be altered by any users. The package check is a read-only process and operates solely within the VAD Registry using read-only functions.
All VAD operations are logged in the server event log. All completed operations are reflected in the DB.DBA.VAD_HISTORY system table.
The optional VADutils package provides some additional administrative tools, mostly for troubleshooting. These include special installation and de-installation functions that can ignore error signals, and provide an interactive editor for the VAD Registry etc.
All operations described below require DBA access to the database.
Check if a VAD package may be installed
DB.DBA.VAD_CHECK_INSTALLABILITY (in package_uri varchar, in is_dav integer); returns varchar
Checks the presence and correct versions of required packages and of the Virtuoso platform. It does not executes any pre-install Virtuoso/PL code from the package, so there's no guarantee that installation will be successful if the check found no error. If package_uri is DAV path, is_dav=1, else is_dav=0.
VAD Package Installation
DB.DBA.VAD_INSTALL (in package_uri varchar, in is_dav integer); returns varchar
If package_uri is DAV path, is_dav=1, else is_dav=0.
The administrator performs the following operations when installing:
The return value of the VAD_INSTALL() function is usually a sum of messages from pre- and post-installation procedures of the package. It should normally contain at least the following:
The VAD packages should be tested to install on an empty Virtuoso database, after any required VAD packages. Installing a package on an empty server is useful for determining that no other procedures or components were missed. Since the application would normally run on the development machine where the VAD package was built, it can be easy to overlook some components. The completeness of the source archive of the application and its independence from any ad hoc SQL objects is important, this is the only way the package can be reliably versioned, tracked or uninstalled.
Check if a VAD package may be uninstalled
DB.DBA.VAD_CHECK_UNINSTALLABILITY (in package_name varchar); returns varchar
Performs a preliminary read-only checks to see whether the package given can be uninstalled. This does not execute any pre-uninstall Virtuoso/PL code from within the package at this stage. Hence, the success of this function does not guarantee that uninstallation will be successful.
VAD Package Uninstallation
DB.DBA.VAD_UNINSTALL (in package_name varchar); returns varchar
The administrator will perform the following operations for the uninstallation process:
Check the state of VAD package installation
DB.DBA.VAD_CHECK (in package_uri varchar); returns varchar
This checks to see if the elements of the package are as they are defined in the original distribution. A list of differing elements is returned. Differences revealed may not indicate a corruption, such changes could have been made intentionally by another package, possibly a later version or upgrade that added some columns to tables, and some resources may be customized by the user post-installation.
This will check for the prior existence of tables, views etc owned by other applications that are not compatible with this application. Any such schema objects found are listed, the installation will not continue. These may be dropped by the DBA to help the installation to succeed. Some such elements may not be part of some other package, hence no package uninstall would be available leading the DBA to drop them with the appropriate SQL commands.
Each package contains full definitions of all tables and indices. Upon installing the following outcomes can occur:
Thus the same SQL schema can be loaded twice without ill effect.
The post install script should be used to populate tables and such. Inserts should be executed using the insert soft statement so that attempts to insert duplicate are silently ignored without causing the installation to fail. The post install script can perform any application level data format changes.
Packages should define their own distinct catalog or qualifier. They should not overwrite another package unless upgrading a prior version. Sometimes a package will require the use of another package's tables. This should be achieved via grants issued in a pre-install script. A schema element such as a table, view or procedure will always have at most one owner package even though it may be referenced or even modified with additional columns by another package installed later. These elements will only be dropped when the owner package is dropped. Tables created ad-hoc from interactive SQL do not have any owner package.
The VAD Sticker contains meta-data and descriptions of resources contained, or to be contained, within a VAD package. Like any XML documents, the target VAD package sticker can be sourced from more than one source file, which can aid maintenance and development.
The namespace vad, used below, represents the URI http://www.openlinksw.com/urn/vad.
The top level element of a VAD Sticker is <sticker>. It must contain a <caption> element and may contain <dependencies>, <procedures>, <ddls> and <resources> elements.
<!--<<top>>--> <!ENTITY % vad.source_sticker "INCLUDE"> <!ENTITY % vad.package_sticker "IGNORE"> <!ENTITY % vad.ecm.group_content "(dependencies | procedures | ddls | resources | registry)" > <![%vad.source_sticker;[ <!ENTITY % vad.ecm.sticker "(caption, (group | %vad.ecm.group_content;)*)"> <!ELEMENT group ((group | %vad.ecm.group_content;)*) > ]]> <![%vad.package_sticker;[ <!ENTITY % vad.ecm.sticker "(caption, %vad.ecm.group_content;)"> ]]> <!ELEMENT sticker %vad.ecm.sticker; > <!ATTLIST sticker version NMTOKEN #REQUIRED xml:lang CDATA #REQUIRED > <!--<</top>>--> <!--<<caption>>--> <!ELEMENT caption (name, version)> <!ELEMENT name ((prop)*)> <!ATTLIST name package NMTOKEN #REQUIRED > <!ELEMENT version ((prop)*)> <!ATTLIST version package NMTOKEN #REQUIRED > <!ELEMENT prop EMPTY> <!ATTLIST prop name NMTOKEN #REQUIRED value CDATA #REQUIRED > <!--<</caption>>-->
The caption contains one name and one version element. These elements have a package attribute for keeping requisites used by VAD procedures. Other prop-s are for keeping admin-readable info, but they will not affect the installer's behavior. Typical names of properties here are Vendor, Copyright, Release+Date, Build, Language, Encoding, but any (even non-unique) names are acceptable.
Sticker's elements for dependencies
<!--<<dependencies>>--> <!ELEMENT dependencies ((require | allow | conflict)*) > <!ATTLIST dependencies> <!ENTITY % vad.ecm.version_list "((version | versions_earlier | versions_later)*)"> <!ELEMENT require (name, %vad.ecm.version_list;) > <!ELEMENT allow (name, %vad.ecm.version_list;) > <!ELEMENT conflict (name, %vad.ecm.version_list;) > <!ATTLIST require group NMTOKEN #IMPLIED > <!ELEMENT versions_earlier ((prop)*)> <!ATTLIST versions_earlier package NMTOKEN #REQUIRED > <!ELEMENT versions_later ((prop)*)> <!ATTLIST versions_later package NMTOKEN #REQUIRED > <!--<</dependencies>>-->
Element dependencies contains an list of packages related to given one. For every version or range of versions of every package, developer may specify whether the given version is required for the package, or allowed but not required, or will cause some sort of troubles.
More precisely, to find information about some particular version of a package, the list of children of dependencies element will be scanned from top to bottom. If the first matching record is in conflict group, not in require or allow, then installation is impossible. From other side, there must be at least one installed package for every require section.
Element require may be labeled with optional group attribute. As an exception from common rule, there must be at least one installed package for every group of require sections with identical name. E.g. If an installation of package B requires either of two interchangeable packages A1 and A2, sticker should contain a pair of nodes in the same group:
<require group="G"> <name package="A1">...</name> </require>
...
<require group="G"> <name package="A2">...</name> </require>
There are no methods to specify that exactly one package, either A1 or A2, should be installed. It must be done by placing proper conflict descriptions in stickers of A1 and/or A2, but not in the sticker of B.
Sticker's elements for procedures
<!--<<procedures>>--> <!ELEMENT procedures ((sql)*)> <!ATTLIST procedures uninstallation (supported | prohibited) #REQUIRED > <![%vad.source_sticker;[ <!ENTITY % vad.sql.include "include CDATA #IMPLIED"> ]]> <![%vad.package_sticker;[ <!ENTITY % vad.sql.include ""> ]]> <!ELEMENT sql (#PCDATA)> <!ATTLIST sql purpose (install-check | pre-install | post-install | uninstall-check | pre-uninstall | post-uninstall) #REQUIRED %vad.sql.include; > <!--<</procedures>>-->
Element procedures contains an list of Virtuoso/PL fragments, and every fragment is tagged by one of four values of the purpose attribute. At every stage of install or uninstall VAD procedure, a whole list of procedures will be scanned from the beginning to the end, and all procedures of appropriate sort will be executed in the same order as they are listed. In source sticker files, include attribute may be used to insert text of some external file instead of having SQL code written inside the element.
Sticker's elements for ddls
<!--<<ddls>>--> <!ELEMENT ddls ((sql)*)> <!ATTLIST ddls > <!--<</ddls>>-->
Element ddls is very similar to procedures and contains an list of Virtuoso/PL fragments to create schemas etc.
Sticker's elements for resources
<!--<<resources>>--> <!ELEMENT resources ((file | location)*)> <!ATTLIST resources > <![%vad.source_sticker;[ <!ENTITY % vad.file.source_uri "source_uri CDATA #IMPLIED"> ]]> <![%vad.package_sticker;[ <!ENTITY % vad.file.source_uri ""> ]]> <!ELEMENT file EMPTY> <!ATTLIST file type (doc | http | dav | code | special) #REQUIRED source (http) "http" target_uri CDATA #REQUIRED makepath (yes | no | abort) "abort" overwrite (yes | no | abort | equal | expected) "equal" package_id CDATA #IMPLIED location IDREF #IMPLIED dav_owner CDATA #IMPLIED dav_grp CDATA #IMPLIED dav_perm CDATA #IMPLIED %vad.file.source_uri; > <!ELEMENT location ((prop)*) > <!ATTLIST location id ID #REQUIRED default_target_uri CDATA #REQUIRED > <!--<</resources>>-->
Element resources lists all files to be copied onto target box. For every file, source and target URIs should be specified, and suggested behavior for cases when a directory should be created or file should be overwritten. Target URI may be relative to one of roots: for documentation, web-resources, DAV, SQL code (it's where virtuoso.ini is located) and one of special locations, additionally specified by location elements. (Installer may query administrator to allow changing of locations' roots; in such case, information from location's properties will be shown to the administrator.) By default, the value of package_id is a space delimited list of type, location ID (if any) and target URI.
To install files into DAV:
<file type="dav" source="http" target_uri="yacutia/yacutia_style.css" dav_owner='dav' dav_grp='administrators' dav_perm='111101101N' makepath="yes"/> <file type="dav" source="http" target_uri="yacutia/yacutia_vdir_style.css" dav_owner='dav' dav_grp='administrators' dav_perm='111101101N' makepath="yes"/>
To install files into file system:
<file type="http" source="http" target_uri="yacutia/yacutia_style.css" makepath="yes"/> <file type="http" source="http" target_uri="yacutia/yacutia_vdir_style.css" makepath="yes"/>
Sticker's elements for registry
<!--<<registry>>--> <!ELEMENT registry ((record)*)> <!ATTLIST registry > <!ELEMENT record ANY> <!ATTLIST record key CDATA #REQUIRED type (STRING | INTEGER | KEY | URL | XML) #REQUIRED overwrite (yes | no | abort | equal | expected) "equal" > <!--<</registry>>-->
Element registry lists all branches to be defined in the VAD Registry. Every record element contain data of one record. The first children of record element (either a text or an element) will be serialized and stored as a value of DB.DBA.VAD_REGISTRY.R_VALUE cell. To prevent errors, it is recommended to keep comments to the data outside the record element: being in the wrong place inside, they may be stored in the registry instead of actually needed data.
A package that contains only some commonly useful ("exported") functions, one table for internal purposes, a small sample VSP application, and small set of documentation files.
<?xml version="1.0" encoding="ASCII" ?> <!DOCTYPE sticker SYSTEM "vad_sticker.dtd"> <sticker version="1.0.010505A" xml:lang="en-UK"> <!-- Name and version; common data about the package --> <caption> <name package="rdf_lib"> <prop name="Title" value="RDF Support Library" /> <prop name="Developer" value="OpenLink Software" /> <prop name="Copyright" value="(C) 2003 OpenLink Software" /> <prop name="Download" value="http://www.openlinksw.com/virtuoso/rdf_lib/download" /> <prop name="Download" value="http://www.openlinksw.co.uk/virtuoso/rdf_lib/download" /> </name> <version package="3.14"> <prop name="Release+Date" value="2003-05-05" /> <prop name="Build" value="Release, optimized" /> </version> </caption> <!-- This package requires no other packages, but it conflicts with package virtodp of versions from 1.00 to 2.17, inclusive --> <dependencies> <allow> <name package="virtodp"></name> <versions_earlier package="1.00"></versions_earlier> </allow> <conflict> <name package="virtodp"> <prop name="Title" value="Virtuoso ODP Sample" /> </name> <versions_earlier package="2.17"> <prop name="Date" value="2001-01-26" /> <prop name="Comment" value="An incompatible version of RDF library is included in some old versions of virtodp " /> </versions_earlier> </conflict> </dependencies> <!-- There are no installation procedures, other than DDLs --> <procedures uninstallation="supported"></procedures> <!-- There are some procedures, which may be re-applying and (maybe) reverted automatically --> <ddls> <sql purpose="pre-install"> "DB"."DBA"."VAD_CREATE_TABLE" ('DB', 'DBA', 'RDF_SCHEDULED_IMPORTS', 'ID integer, URI varchar, CALLBACK varchar, VERSION varchar, REPORT long varchar, primary key (ID)'); </sql> <sql purpose="post-install"> "DB"."DBA"."VAD_LOAD_RESOURCE" ('rdf_functions'); </sql> </ddls> <!-- Resources include... --> <resources> <!-- ...documentation, ... --> <file type="doc" target_uri="rdf_lib/1.1/intro.dxt" /> <file type="doc" target_uri="rdf_lib/1.1/interface.dxt" /> <file type="doc" target_uri="rdf_lib/1.1/implementation.dxt" /> <file type="doc" target_uri="rdf_lib/1.1/sample_app.dxt" /> <!-- ...the file of commonly-useful functions, ... --> <file package_id="rdf_functions" type="code" target_uri="rdf_lib/1.1/rdf_lib.sql" /> <!-- ...pages of the sample application, named rdf_edit, ... --> <file type="http" target_uri="rdf_lib/rdf_edit/default.htm" /> <file type="http" target_uri="rdf_lib/rdf_edit/browse.vsp" /> <file type="http" target_uri="rdf_lib/rdf_edit/edit.vsp" /> <!-- ...a DAV resource with sample RDF data, ... --> <file type="dav" target_uri="rdf_lib/sample_odp_structure.rdf" /> <!-- ...two files of sample application's functions. --> <file type="code" target_uri="rdf_lib/1.1/rdf_edit/content_level.sql" /> <file type="code" target_uri="rdf_lib/1.1/rdf_edit/view_level.sql" /> </resources> <!-- There are no application-specific registry items in this package --> </sticker>
Administering a database involves taking backups and having a readiness to recover from backups and subsequent transaction logs.
Backups can be taken in two principal ways:
The Virtuoso backup functions can be used from any client directly, such as ISQL. It is possible, and perhaps preferable, to create stored procedures for performing the backup functions and scheduling these with the Virtuoso scheduler.
The actual database file(s) can be copied while the database is running so long as no checkpoint is made during the copy process. Checkpointing can be disabled for this, but make sure it is re-enabled after the backup has been made.
Making a full backup of a large database can take several hours if not days just due to the speed of tapes or local area networks. Full backups must in all cases be done without an intervening checkpoint. This is why frequent full backups are not desirable. To ensure the possibility of full recovery one must have the complete set of transaction logs (audit trail log) since the last backup.
Restarting the database after restoring backed up files will show the state in effect since the last checkpoint preceding the backup. Any transaction log files made after the point of backup can be replayed to bring the state up to the last recorded transaction.
Virtuoso can maintain a transaction audit trail. This is enabled using the CheckpointAuditTrail setting in the virtuoso.ini file. When this setting is non-zero, Virtuoso will begin a new transaction log after each checkpoint. Thus one automatically gets a full, unbroken sequence of transaction logs for the entire age of the database. These logs are named as specified in virtuoso.ini and are suffixed with their creation time.
Transaction logs older than the log that was current at the time of the last backup are superfluous for recovery, since their transactions were checkpointed before the backup started. Transactions of the log current at the time of the backup are NOT in the backed up state since they were not checkpointed, i.e. written into the read-only section of the database containing the data being backed up.
We strongly advise having the CheckpointAuditTrail enabled in any production environment.
It is good practice to have at least two generations of full backup, since the last backup may contain errors that were not known at the time of its making. If such precaution is taken then only transactions logs older than the oldest backup are safe to remove. If we needed to recover from the oldest backup for any reason we would require all audit transaction logs created during and after that backup.
A Virtuoso database can be restored from the last full backup and all Audit Trail transaction files created during and after the backup. You would need to start the database as normal with the backup version of the database file. Once the database has been started, connect using iSQL. You can then use the replay() function to replay the transaction files up to the required point. It is vital that these files are replayed in the correct order.
Virtuoso is capable of performing online backups so that normal database operation does not have to be disrupted in order to take backups. The backup_online() can be used to backup the database in the state effective at the last checkpoint to a series of backup files.
backup_online ( file_prefix , pages , [timeout], [dirst]);
The database storage is divided into a checkpoint space that is a read only image from the time of the last checkpoint and thus can be safely backed up anytime between checkpoints and the commit space where updates subsequent to the last checkpoint data are stored. Additionally, the database records what pages have changed since the last checkpoint every time new checkpoint is made. This change tracking makes it possible to make incremental backups. The first time the backup_online function is called, it saves a compressed copy of the then current checkpoint state into one or more files. The next time it is called, it will write the changes that have come into the checkpoint space since the last time backup_online was called. It is possible to erase the change tracking data with the backup_context_clear function. The next call to backup_online will then make a full backup. Files generated by one or more calls to backup_online without intervening backup_context_clear form a series with distinct serial numbers and will be restored together. In order to restore such files, the administrator must delete the previous database files and start the server with a special flag and indicate the location of the backup files. This will bring the database to the state corresponding to the state as of the checkpoint immediately preceding the last call to backup_online, i.e. the one that wrote the newest of the backup files being restored. To restore onwards from this state, the administrator must replay transaction logs, starting with the log that was current when the last call to backup_online was made. In order to preserve all such logs, one must run with the CheckpointAuditTrail ini parameter set to 1.
A database checkpoint cannot be performed while an online backup is in progress.
SQL> backup_context_clear (); SQL> checkpoint; SQL> backup_online ('virt-inc_dump_#', 150);
The backup_online() procedure differs from the the CheckPointAuditTrail mainly because it can be started from any point in the database. Unless CheckPointAuditTrail was enabled when the database was created, the database file at a particular state and all transaction logs created by the AuditTrail since that state would be required to restore the database. Only the backup set files would be required to restore from backup_online(). The backup_online() also makes a compressed backup, making it far more suitable for large databases.
The last optional parameter allows to point the directory(ies) where the backup files must be stored. See backup_online() description for details.
To restore from a backup series the administrator must first shutdown the Virtuoso database server and move all database files (e.g. virtuoso.db and virtuoso.trx) out of the database directory. It is recommended that copies by taken rather than deleting them entirely. Then the command:
<virtuoso exe> +restore-backup <FILE_PREFIX>
-- for example: virtuoso-iodbc-t +restore-backup dump-20011010_#
must be issued in the directory where the "*.bp" were stored. The database will then be restored. The expression <virtuoso exe> above must be replaced with the path and filename to the Virtuoso server executable used on your system (e.g. ..\virtuoso-odbc-t.exe).
Each file in the series has a header containing a unique identifier, for the backup set and the sequence number of the file in the backup set . If an identifier in any file in the backup sequence differs from the identifier contained in the first file, the restoration process will stop and report an error, which is written to the Virtuoso log file.
At times the backup or restoration commands may return errors. Use the following list to help diagnose and resolve them:
The unique identifier in header of the file differs from the identifier of the first file. It is possible that the file was renamed or corrupt or belongs to another backup set.
The sequence number of the file does not correspond to the internal sequence number of file. This could be caused by the file being renamed or corrupt.
The prefix of the file does not correspond to internal information. Possible reason: file was renamed or corrupt or belongs to another backup set.
Read error. Possible reason: Virtuoso database file was corrupt.
Read error. Possible reason: Virtuoso database file was truncated or hardware error.
Read error. Possible reason: Virtuoso database file was truncated or hardware error.
Write error. Possible reason: disk is malfunctioning or full.
Following the online backup example above:
SQL> backup_context_clear (); SQL> checkpoint; SQL> backup_online ('virt-inc_dump_#', 150);
The following command could be used to restore the database from the backup files created:
virtuoso-iodbc-t -c <db-ini-file> +restore-backup virt-inc_dump_#
or:
virtuoso-odbc-t.exe -c <db-ini-file> +restore-backup virt-inc_dump_#
A possible way of making a full backup of a large databases is first to turn off any automatic checkpoints and make a compressed copy of the files. After the back up is complete, checkpointing should be re-enabled. The files should be compressed to make efficient use of space, and should be copied to a disk separate from the location of the database, and preferably to an external backup medium such as tape.
For a large database it is best to turn off any automatic checkpoints and copy the database files to external storage. Checkpoints should be turned off by issuing the command:
checkpoint_interval (-1);
at the SQL prompt. Checkpoints can be re-enabled in a post-backup script by:
checkpoint_interval (<n>);
which sets the automatic checkpoint interval to <n> minutes. The backup will be unusable if there are checkpoints made while it is in progress. Thus it is important to guarantee that checkpoints do not occur. The only safe way of doing this is the above, since it is in principle possible to have a server crash during the backup and a roll forward following restart, all while the backup is in progress. If this happens the backup will be readable and consistent with the state of the last checkpoint if and only if there are no checkpoints between its start and completion. Setting the interval to -1 will guarantee that the server, when starting after recovery will not make a checkpoint.
The dba must make sure that clients do not issue checkpoint or shutdown statements while a backup is in progress.
The presence or absence of checkpoints at a given point in time can be ascertained from the virtuoso.log event log file.
When Virtuoso is not running a complete and clean backup can be taken by making a copy of the database file and transaction file(s) created after the last checkpoint.
To get an up to the minute copy of a running database one can copy the database file and the associated log, i.e. the file specified in TransactionFile in the database's configuration file. When started, the log will roll forward and restore the database to the state following the last logged transaction.
The process of rebuilding a database consists of dumping its contents into a large log file, or log files, and doing a roll forward from an empty database with that log.
The general steps to rebuild a database are:
Shut down the running server, making a checkpoint. This is done with the SHUTDOWN command from interactive SQL.
Make sure there is a log file specified in virtuoso.ini.
Start the server process virtuoso with the -b command line option: e.g. % ./virtuoso -b (+backup_dump)
This will write the contents of the database into the log file specified in virtuoso.ini and exit when complete.
Take a backup of the old database file.
Change the DatabaseFile setting in virtuoso.ini to a non-existing database file or delete the old database file.
Start the server with the +restore-crash-dump option. The option is essential.
The server will build a new database file from the log and once completed it will, by default, perform a checkpoint of the transactions to the database file and start listening at the specified port. Virtuoso can be started without a checkpoint using the -n (+no-checkpoint) option.
You may then connect to the database with interactive SQL and if necessary make a checkpoint. The checkpoint will write freeze the state following roll forward and delete the log used for the rebuild.
The database is now ready for normal use.
It is recommended you take a backup copy of the database file(s)prior to this procedure.
It may sometimes be useful to rebuild a database as above to save space. Virtuoso does not relinquish space in the DB file back to the file system as records are removed, however, Virtuoso does reuse pages that are made available from a deletion of records. The steps above will build a new compact database file. You would ordinarily not have to worry about this.
It is possible to recover data from a damaged Virtuoso database by a procedure similar to rebuilding a database as described above. A database file may be corrupt if the database repeatedly crashes during a specific operation.
To determine whether a database is corrupt, you may use the backup to a null file command in isql, for Unix platforms:
SQL> backup '/dev/null';
For windows platforms you can use:
SQL> backup 'NUL';
This command will read through the database checking its integrity. If the server crashes before completing the backup process, then the database is indeed corrupt and needs to be recovered. No other activity should take place while the command is executing.
To recover the database, follow the procedure for rebuilding it, except use the -D 'capital D' or +crash-dump switch instead of -b. This will construct a log file which you can replay to make a new database. The database will contain the transactions that were committed as of the last successful checkpoint. If the database altogether fails to open it may be the case that the schema is damaged.
It is possible that the database to be recovered is too large to fit in a single log file. The crash dump feature therefore allows segmenting the recovery log into a number of files. See the virtuoso.ini configuration file documentation for details. It is possible to make a crash dump in several pieces if there is not enough total disk space to hold the dump on the system where the database is running.
If the recovery log is split over several files it is necessary to set the transaction file in the ini to point to the first of these files, delete the database file(s) and start the server with the +restore-crash-dump option. When the server comes online, one can connect to it with isql and use the replay () function for replaying the remaining logs, one by one, in their original order.
For example,assuming the virtuoso.ini fragment:
Log1 = rec-1.log 100M Log2 = rec-2.log 100M
we would make the dump with
virtuoso +crash-dump
and once the server has been started with +restore-crash-dump, with the ini setting TransactionFile set to rec1.log, replay the remaining log with the isql commands:
SQL> replay ('rec-2.log'); Done. SQL> checkpoint;
If the recovery is interrupted it can be restarted at the last checkpoint made during the recovery. Note that a mid recovery checkpoint may take a very long time, e.g. 1 hour for a 10GB database, since it is possible that the delta since the previous recovery checkpoint comprise almost all the database.
When the schema tables (e.g. DB.DBA.SYS_COLS, DB.DBA.SYS_KEYS) have corrupt rows the normal crash dump/crash restore procedure will not be possible because the server relies on the schema tables to know the key layouts for reading the data rows of other tables upon crash dump.
In such situations there is a special procedure to be followed to save as much data as possible from the corrupt database. The general steps are:
Thus the transaction log produced from the corrupt database, when replayed on the new database file (the one holding the schema tables data) makes the closest approximation to the corrupt database's data. However, this will not produce a workable database by itself - it may possibly deny inserting of data into tables with IDENTITY columns and will lose all the data within the Virtuoso registry (accessible from registry_get()/registry_set() functions).
Because of the very nature of the crash-restore process described here and because of the fact that data is lost in the database schema, the server will not attempt to dump tables whose schema description is lost. So care should be taken when reading the data from the database.
This restoration procedure in no way replaces the regular database backup procedures, it merely tries to save whatever reasonable data there may be left from the database file.
The recovery sequence is as follows:
Do a crash dump of the schema tables (using the '+crash-dump +mode oa +dumpkeys schema' virtuoso command line options).
Create a new INI file to describe the layout of the new database you'll use to temporarily fill up the restored data.
Move the transaction log file(s) produced in step 1 to the location required by the new INI file.
Replay the transaction log from step 1 on an empty database using the new INI file. You will now have the schema tables readable in the new database (and nothing else):
Virtuoso options : -c <your new ini file> +restore-crash-dump -f
Make a crashdump of the data in the non-schema tables of the old database while having read the schema tables from the new database:
+crash-dump -c <your new ini file> +crash-dump-data-ini <your old ini file> +mode o -f
Move the transaction log file(s) produced in step 1 to the location required by the new INI file.
Replay the transaction log from the previous step into the new db file using:
-c <your new ini file> +restore-crash-dump
Do a normal crash dump of the new database:
-c <your new ini file> +crash-dump
Move away (backup) the original (old) database files and put the transaction log produced by the above step into the location specified in the original INI file. You can also delete the rest of the DB files of the new database at that point.
Replay the transaction log to make the old database afresh.
To automate the above procedure, a sample Unix script follows that automates it somewhat. This script expects the crashed database virtuoso.db.save and an appropriate INI file (no striping, no log segmentation, transaction log file name virtuoso.trx) in the current directory and creates the restored database. It also expects the virtuoso-iodbc-t executable to be in the operating system path. Also, make sure that you have a suitable virtuoso.lic license file in the current directory.
#!/bin/sh rm -rf xmemdump.txt virtuoso.trx virtuoso.tdb virtuoso.log virtuoso.db virtuoso.lck core.* new.lck new.log new.trx new.tdb new.db new.ini cp -f virtuoso.db.save virtuoso.db cat virtuoso.ini | sed 's/virtuoso\./new./g' > new.ini virtuoso-iodbc-t -f +crash-dump +mode oa +dumpkeys schema ls -la *.trx mv virtuoso.trx new.trx virtuoso-iodbc-t -c new -f -R virtuoso-iodbc-t -c new +crash-dump +crash-dump-data-ini virtuoso.ini +mode o -f ls -la *.trx mv virtuoso.trx new.trx virtuoso-iodbc-t -c new -R -f virtuoso-iodbc-t -c new +crash-dump -f rm -f virtuoso.trx virtuoso.tdb virtuoso.log virtuoso.db virtuoso.lck ls -la *.trx mv new.trx virtuoso.trx virtuoso-iodbc-t -R -f
If the database was created with a version prior to the one being used for rebuilding, the system tables may be different. The creation here refers to the first time the database was made, a crash recovery does not count as a fresh start here.
If this is or may be the case, the first log must be rolled forward into the empty database BEFORE the new and possibly incompatible system tables are created. This is done by setting the TransactionFile parameter to the first of the recovery logs and starting the server with the -R or +restore-crash-dump switch. For good practice one should also specify the no checkpoint switch, so that the log will in no case be damaged after the initial step of the roll forward. After this initial step the system tables will be compatible and the dba can proceed to replay the remaining recovery logs with the replay function.
This section provides a checklist for improving performance of a Virtuoso server.
If something does not work fast, this is mostly for the following reasons:
Determining which is the case is simple. The result set returned by status ('') has most of the information. Do this twice with some 10 seconds between the samples and see the second result set.
If there is not enough memory, there will be frequent disk access. This is seen from the buffers and disk usage lines.
The very simplest test for this is looking at the CPU % of the process in top. If there is constant load and the percent is low then the server is IO bound.
If all memory is not in use, then memory cannot be the problem. This is seen from the buffers line. If the used number is under 80% of the total or if the replace age is several times larger than the count of buffers, then things are OK. If the replace age is 0 then no buffers have ever been replaced and all that ever was read is still in memory.
If the replace age is less than or close to the buffer count times 4, then cache replacement is frequent and adding buffers is advised.
Adding more than 60-70% of system ram as buffers is not useful. The setting is NumberOfBuffers in the ini file, count 9K per buffer.
The disk access is summarized on the disk usage line. First is the number of reads since start then the average latency in N ms. If %r is high, then a lot of the time between the previous status and this is taken by disk. This can be over 100. One thread that is waiting for disk all the time counts for 100. If the percent is high then adding more disks and striping over them will be useful. Even with a single database file, adding file descriptors (FDsPerFile setting in the ini) may be useful. If the average read latency is 0 or close, then the data is cached by the OS. If it is high, then adding disks and striping may reduce it.
The read ahead line will tell if there are sequential reads. These are faster than random ones and can efficiently use striped disks.
If the workload is random access, then a high number of read ahead means that one might not have the required indices, thereby causing full table scans. More on this in the query plans section below.
Swapping is always bad. If swapping occurs, then one has too many buffers and should decrease the number of buffers. Use an OS tool like top to see the size of the database process and its virtual memory use. Having a resident size smaller than logical size is not always bad since some code or data in the process may simply be unused but having, after running in a steady state, i.e. all buffers used, a resident size less than the amount of memory allocated for buffers is always bad. Before steady state, i.e. during cache warm-up, the resident size is normally less than the buffer pool's size.
To see the count of major page faults. i.e. ones that read the disk, do:
select getrusage ()[]4];
through interactive SQL. The result is the count of major faults since starting the process. The count should not vary between samples, at least not more than a few units per minute.
This function is not available under Windows. Use the task manager instead for tracking this.
There can be an actual memory leak, specially with plugins or hosting. See the growth of the virtual size throughout the run, reaching full buffer utilization and doing a couple of checkpoints. Past this, the process should not grow. You may also see if the ThreadCleanupInterval or ResourcesCleanupInterval ini parameters have an effect. If in spite of all the process grows there can be a genuine leak. It is normal for the size to fluctuate somewhat due to varying amounts of uncommitted data, threads, connections and the like.
The lock status section indicates the count of deadlocks and waits. If the 2r1w number is high then it means that the application should read for update. It gets shared locks and cannot change them to exclusive. Adding the for update option to selects in the right place will fix this, also setting DefaultIsolation in the ini to 2 for read committed will be good. If the lock wait count increases fast, then locking may downgrade performance. The threads line below shows how many threads have some task to do. The waiting is the number of these threads that are waiting for a lock at the time of the status. The vdb number is the number of threads waiting for network io, either from virtual database or any web protocol.
Take a few samples and if these show few threads waiting and the waits or deadlocks counts do not climb much locking will not be a problem. If these numbers are high, see the sys_l_stat view for details.
select sum (waits), sum (wait_msecs) from sys_l_stat;
for the totals. The first number is the count of times a thread waited for a lock, the second is the sum of the real times spent waiting.
select top 10 * from sys_l_stat order by waits desc;
shows the keys on which most waits take place. See also the deadlocks and the wait_msecs columns. Numbers are cumulative.
For details on disk, see:
select top 10 * from sys_d_stat order by reads desc;
These views are explained in more detail in the performance meters section.
If there is a multi-user application doing random access for read or write, it is an idea to partition the data so that they do not hit the same page all the time. For example, to allow for parallel insert of orders without contention, the TPC C schema prefixes the order number with a warehouse and district number, so that simultaneous inserts will most often not hit the same page.
Even when there is no locking, there is still some serialization for database page access. This is shown in the various wait columns of SYS_L_STAT. If the sum of the waits for a key is over 1% of the touches for the key, as given in SYS_D_STAT, there is contention and a performance penalty of maybe 10%.
Such things can be improved by altering the schema design and configuration parameters do not usually help. If there is disk access, having more memory always helps because then locks will be in effect for less time.
To know if there is a bad query plan, see the explain output of the query.
Unless a full table scan is intended, a full table scan is pretty much always bad.
In the explain () result set, a table access is marked by a From table name. Below is mentioned the driving index, the conditions on the index and the conditions that are not indexable.
Joins are listed one after the other, the outermost first. The join order can be read from the explain output going from top to bottom.
If a table is read on a non primary key and columns not covered by the driving index are accessed, there is a second key mentioned with a full match of the primary key columns as condition.
A full table scan looks like this:
explain (select count (*) from t1'); ... from DB.DBA.T1 by STR2 8.4e+04 rows Key STR2 ASC () Local Code 0: $30 "count" := artm $30 "count" + <constant (1)> ...
There are no conditions mentioned.
A lookup with index looks like this.
explain ('select fi2 from t1 where row_no = 11'); ... from DB.DBA.T1 by T1 Unique Key T1 ASC ($26 ".FI2") inlined <col=1694 ROW_NO = <constant (11)>> ...
The condition is shown on the line below the key.
A lookup with full scan testing every row looks like this:
... from DB.DBA.T1 by T1 2.5e+04 rows Key T1 ASC ($26 ".FI2") row specs: <col=1699 FI2 > <constant (11)>> ...
The condition is shown after the heading row specs. The whole key mentioned in the key will be read and the entries tested. if both indexed and row tests exist, the indexed are done first as one would think.
If your query has full table scans, consider adding an index.
If the index choice is not the right one, consider the following possibilities:
Hash joins can make full table scans. This is OK if the table scanned is small.
A hash join looks like this:
explain ('select count (*) from t1 a, t1 b where a.row_no = b.row_no + 1'); ... { Fork { Fork { from DB.DBA.T1 by STR2 8.4e+04 rows Key STR2 ASC ($26 "B.ROW_NO") Local Code 0: $30 "temp" := artm $26 "B.ROW_NO" + <constant (1)> 4: BReturn 0 Current of: <$28 "<DB.DBA.T1 B>" spec 5> Sort (HASH) ($30 "temp") -> ($26 "B.ROW_NO") } from DB.DBA.T1 by STR2 8.4e+04 rows Key STR2 ASC ($37 "A.ROW_NO") Current of: <$39 "<DB.DBA.T1 A>" spec 5> Hash source ($37 "A.ROW_NO") -> ($26 "B.ROW_NO") After code: 0: $44 "count" := artm $44 "count" + <constant (1)> 4: BReturn 0 } Select ($44 "count", <$39 "<DB.DBA.T1 A>" spec 5>, <$28 "<DB.DBA.T1 B>" spec 5>) }
First t1 is read from start to end and a hash is filled with row_no + 1. Then t1 is read from start to end a second time. The hash source is the hash lookup. A hash join, if there is no index or if the whole table or a large part thereof is traversed is better than a loop join because it replaces random access with sequential. The complexity is close to O(n + n) instead of O (n * log n). Plus sequential read makes better use of read ahead.
The corresponding loop join looks like this:
explain ('select count (*) from t1 a, t1 b where a.row_no = b.row_no + 1 option (loop)'); { Fork { from DB.DBA.T1 by STR2 8.4e+04 rows Key STR2 ASC ($26 "B.ROW_NO") Current of: <$28 "<DB.DBA.T1 B>" spec 5> Precode: 0: $29 "temp" := artm $26 "B.ROW_NO" + <constant (1)> 4: BReturn 0 from DB.DBA.T1 by T1 Unique Key T1 ASC () inlined <col=1694 ROW_NO = $29 "temp"> Local Code 0: $35 "count" := artm $35 "count" + <constant (1)> 4: BReturn 0 Current of: <$31 "<DB.DBA.T1 A>" spec 4> } Select ($35 "count", <$31 "<DB.DBA.T1 A>" spec 4>, <$28 "<DB.DBA.T1 B>" spec 5>) }
To prevent hash joins, use table option or option at the end of the select as seen above. A hash join is very bad if a whole table is read for filling a hash and then only a small number of entries are fetched from the hash. However, if there is no index, then even this is better than a loop join.
A checkpoint can take a very long time in certain special conditions. Normally a log checkpoint is about a minute if flushing several G worth of buffers to disk. The flushing is mostly done online, after which there is an atomic time of a few seconds. After this operation resumes. Applications do not notice the checkpoint except as a temporary increase in response delays.
If spikes in response time are to be avoided, then making frequent checkpoints is better than making infrequent ones.
If there are long running transactions with many locks or uncommitted changes, then the checkpoint interval should anyhow be longer than several times the expected duration of such a transaction. If this is not so, the checkpoint can fall in the middle of the transaction and will internally have to rollback and again reestablish the uncommitted state so as to write a clean image on the new checkpoint. Doing this many times in the life of a transaction is very inefficient. In generally do not make long transactions with locking. Preferentially use read committed for anything long.
A checkpoint's atomic time can be prohibitively long under the following circumstances:
Virtuoso allows splitting a database over several files that may be on different devices. By allocating database fragments onto independent disks I/O performance in both random and sequential database operations can be greatly enhanced.
The basic unit of a database is the segment. A segment consists of an integer number of 8K pages. A segment may consist of one or more files called stripes. If a segment has multiple stripes these will be of identical length and the segment size will be an integer multiple of the stripe size.
The size limit on individual database files is platform dependent, but 64 bit file offsets are used where available. For large databases use of multiple disks and segments is recommended for reasons of parallelism even though a single database file can get very large. A database can in principle grow up to 32TB (32-bit page number with 8KB per page).
When a segment is striped each logically consecutive page resides in a different file, thus for a segment of 2 stripes the first stripe will contain all even numbered pages and the second all the odd numbered pages. The stripes of a segment should always be located on independent disks.
In serving multiple clients that do random access to tables the server can perform multiple disk operations in parallel, taking advantage of the independent devices. Striping guarantees a statistically uniform access frequency to all devices holding stripes of a segment.
The random access advantages of striping are available without any specific configuring besides that of the stripes themselves.
Striping is also useful for a single client doing long sequential read operations. The server can detect the serial nature of an operation, for example a count of all rows in a table and can intelligently prefetch rows.
If the table is located in a striped segment then the server will read all relevant disks in parallel if these disks are allocated to different I/O queues.
All stripes of different segments on one device should form an I/O queue. The idea is that the database stripes that benefit from being sequentially read form a separate queue. All queues are then read and written independently, each on a different thread. This means that a thread will be allocated per queue. If no queues are declared all database files, even if located on different disks share one queue.
A queue is declared in the striping section by specifying a stripe id after the path of the file, separated by an equal sign.
[Striping] Segment1 = 200M, disk1/db-seg1-1.db = iq1, disk2/db-seg1-2.db = iq2 Segment2 = 200M, disk1/db-seg2-1.db = iq1, (disk2/db-seg2-2.db = iq2
In the above example the first stripes of the segments form one queue and the second stripes form another. This makes sense because now all database files on /disk1 are in iq1 and all on /disk2 are on iq2.
This configuration could have resulted from originally planning a 200 MB database split on 2 disks and subsequently expanding that by another 200 MB.
The I/O queue identifier can be an arbitrary string. As many background I/O threads will be made as there are distinct I/O queues.
Striping and using I/O queues can multiply sequential read rates by a factor almost equal to the number of independent disks. On the other hand assigning stripes on one disk to different queues can have a very detrimental effect. The rule is that all that is physically accessed sequentially will be on the same queue.
One should keep the following in mind when designing a schema for maximum efficiency.
A select from a table using a non-primary key will need to retrieve the main row if there are search criteria on columns appearing on the main row or output columns that have to be fetched from the main row. Operations are noticeably faster if they can be completed without fetching the main row if the driving key is a non-primary key. This is the case when search criteria and output columns are on the secondary key parts or primary key parts. Note that all secondary keys contain a copy of the primary key. For this purpose it may be useful to add trailing key parts to a secondary key. Indeed, a secondary key can hold all the columns of a row as trailing key parts. This slows insert and update but makes reference to the main row unnecessary when selecting using the secondary key.
A sequential read on the primary key is always fastest. A sequential search with few hits can be faster on a secondary key if the criterion can be evaluated without reference to the main row. This is because a short key can have more entries per page.
Each column takes the space 'naturally' required by its value. No field lengths are preallocated. Space consumption for columns is the following:
Data | Bytes |
---|---|
Integer below 128 | 1 |
Smallint | 2 |
long | 4 |
float | 4 |
timestamp | 10 |
double | 8 |
string | 2 + characters |
NULL | data length for fixed length column, as value of 0 length for variable length column. |
BLOB | 88 on row + n x 8K (see note below) |
If a BLOB fits in the remaining free bytes on a row after non-LOBs are stored, it is stored inline and consumes only 3 bytes + BLOB length.
Each index entry has an overhead of 4 bytes. This applies to the primary key as well as any other keys. The length of the concatenation of the key parts is added to this. For the primary key the length of all columns are summed. For any other key the lengths of the key parts plus any primary key parts not on the secondary key are summed. The maximum length of a row is 4076 bytes.
In light of these points primary keys should generally be short.
For data inserted in random order pages tend to be 3/4 full. For data inserted in ascending order pages will be about 90% full due to a different splitting point for a history of rising inserts.
Virtuoso offers an execution profiling mechanism that keeps track of the relative time consumption and response times of different SQL statements.
Profiling can be turned on or off with the prof_enable function. When profiling is on, the real time between the start and end of each SQL statement execute call is logged on the server. When prof_enable is called for the second time the statistics gathered between the last call to prof_enable and this call are dumped to the virtprof.out file in the server's working directory.
Profiling is off by default. Profiling can be turned on with the statement:
prof_enable (1);
The virtprof.out file will be generated when prof_enable is called for the second time, e.g.
prof_enable (0);
will write the file and turn profiling back off.
Below is a sample profile output file:
Query Profile (msec) Real 183685, client wait 2099294, avg conc 11.428772 n_execs 26148 avg exec 80 99 % under 1 s 99 % under 2 s 99 % under 5 s 100 % under 10 s 100 % under 30 s 23 stmts compiled 26 msec, 99 % prepared reused. % total n-times n-errors 49 % 1041791 7952 53 new_order 28 % 602789 8374 490 delivery_1 12 % 259833 8203 296 payment 5 % 123162 821 35 slevel 2 % 54182 785 0 ostat (?, ?, ?, ?) 0 % 11614 4 0 checkpoint 0 % 2790 2 1 select no_d_id, count (*) from new_orde 0 % 2457 3 1 select count (*) from new_order 0 % 662 2 0 status () 0 % 11 1 1 set autocommit on 0 % 3 1 0 select * from district
This file was produced by profiling the TPC C sample application for 3 minutes. The numbers have the following meanings:
The real time is the real time interval of the measurement, that is the space in time between the prof_enable call that started the profiling and the call that wrote the report. The client wait time is the time cumulatively spent inside the execute call server side, only calls completely processed between profiling start and end are counted. The average concurrency is the exec time divided by real time and indicates how many requests were on the average concurrently pending during the measurement interval. The count of executes and their average duration is also shown.
The next section shows the percentage of executes that finished under 1, 2, 5, 10 and 30 seconds of real time.
The compilation section indicates how many statements were compiled during the interval. These will be SQLPrepare calls or SQLExecDirect calls where the text of the statement was not previously known to the server. The real time spent compiling the statements is added up. The percentage of prepared statement reuses, that is, executes not involving compiling over all executes is shown. This is an important performance metric, considering that it is always better to use prepared statements with parameters than statements with parameters as literals.
The next section shows individual statements executed during the measurement interval sorted by descending cumulative real time between start and completion of the execute call.
The table shows the percentage of real time spent during the calls to the statement as a percentage of the total real time spent in all execute calls. Note that these real times can be higher than the measurement interval since real times on all threads are added up.
The second column shows the total execute real time, the third column shows the count of executes of the statement during the measurement interval. The fourth column is the count of executes that resulted in an error. The error count can be used for instance to spot statements that often produce deadlocks.
Statements are distinguished for profiling purposes using the 40 first characters of their text. Two distinct statements that do not differ in their first 40 characters will be considered the same for profiling. If a statement is a procedure call then only the name of the procedure will be considered, not possibly differing literal parameters.
The profiler will automatically write the report after having 10000 distinct statements in the measurement interval. This is done so as to have a maximum on the profiling memory consumption for applications that always compile a new statement with different literals, resulting in a potentially infinitely long list of statements each executed once. It is obvious that such a profile will not be very useful.
It cannot be overemphasized that if an application does any sort of repetitive processing then this should be done with prepared statements and parameters, for both reasons of performance and profilability.
Note that measurements are made with a one millisecond precision. Percentages are rounded to 2 digits. Timing of fast operations, under a few milliseconds, will be imprecise as a result of the 1 ms resolution. Also the cumulative compilation time may be substantially off, since the compilation may often take less than 1 ms at a time. Also note that the precision may also vary between platforms.
These views provide statistics on the database engine
create view SYS_K_STAT as select KEY_TABLE, name_part (KEY_NAME, 2) as index_name, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'n_landings') as landed, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'total_last_page_hits') as consec, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'page_end_inserts') as right_edge, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'page_end_inserts') as lock_esc from SYS_KEYS;
create view SYS_L_STAT as select KEY_TABLE, name_part (KEY_NAME, 2) as index_name, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_set') as locks, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_waits') as waits, (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_waits') * 100) / (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_set') + 1) as wait_pct, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'deadlocks') as deadlocks, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'lock_escalations') as lock_esc from SYS_KEYS;
create view sys_d_stat as select KEY_TABLE, name_part (KEY_NAME, 2) as index_name, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'touches') as touches, key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'reads') as reads, (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'reads') * 100) > / (key_stat (KEY_TABLE, name_part (KEY_NAME, 2), 'touches') + 1) as read_pct from SYS_KEYS;
These views offer detailed statistics on index access locality, lock contention and disk usage.
'reset' specified as the stat name will reset all counts for the key in question.
KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES
INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.
LANDED The count of random accesses, including inserts. Any insert or select, whether empty, single line or multi- line counts as one. Updates and deletes do not count, as they imply a select in the same or previous statement.
CONSEC The number of times a random access falls on the same page as the previous random access. This is always less than LANDED. For repetitive access to the same place or an ascending insert, this will be near LANDED. For a totally random access pattern this will be near 0.
RIGHT_EDGE The number of times an insert has added a row to the right edge of the page where the insert was made.
LOCK_ESC The count of lock escalations, see SYS_L_STAT.
KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES
INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.
LOCKS The number of times a lock has been set on the index. Making a new row or page lock counts as one. Entering a row or page lock either after a wait or without wait (for a shared lock) counts as one.
WAITS The number of times a cursor reading this index waited for a lock. Note that this can be higher than the number of locks set, e.g. a 'read committed' cursor may wait for a lock but will never make one.
WAIT_PCT The percentage of lock set events that involved a wait.
DEADLOCKS The number of times a deadlock was detected when trying to wait for a lock on this index. Note that one deadlock event may involve locks on several indices. Each deadlock detection counts as one.
LOCK_ESC The number of times the set of row locks on a page of this index where escalated into one page lock or a page lock was set initially. This is always less than LOCKS. This value will be near LOCKS when there are many sequential selects which switch to page lock mode. This happens when a cursor has performed over 2 lock escalations and the page being entered has no locks, i.e. the lock can be set over the entire page.
KEY_TABLE The fully qualified table name, e.g. DB.DBA.SYS_PROCEDURES
INDEX_NAME The name of the index. This will be equal to the table name for the table's primary key.
TOUCHES The number of times a row is located on the index. Every row retrieved by a select or inserted counts as one. All rows scanned by an select count or other aggregate counts as one.
READS The number of times a disk read was caused by a read operation on this index. This may theoretically be higher than TOUCHES, since several levels of the index tree may have to be read to get to a leaf.
READ_PCT The percentage of READS in TOUCHES.
select index_name, locks, waits, wait_pct, deadlocks from sys_l_stat order by 2 desc; Get lock data, indices in descending order of lock count. select index_name, touches, reads, read_pct from sys_d_stat order by 3 desc; Get disk read counts, index with most reads first. select index_name, (consec * 100) / (landed + 1) from sys_k_stat where landed > 1000 order by 2;
Get the percentage of consecutive page access on indices with over 1000 accesses so far, most randomly accessed first.
This function returns a summary of the database status as a result set. The result set has one varchar column, which has consecutive lines of text. The lines can be up to several hundred characters.
The contents of the status summary are described in the Administrator's Guide.
All data in a virtuoso database are logically stored as database key rows. Thus the primary key for a table holds the entire row (including the dependent part) and the secondary keys just hold their respective key parts. So the space that the table occupies is the sum of the space occupied by it's primary key and all the secondary keys.
The main physical unit of allocation in a virtuoso db file is the database page (about 8k in virtuoso 3.x). So the server needs to map the key rows and outline blobs to database pages.
Virtuoso will store as many rows in a db page as it can, so usually one DB page will contain more than 1 row of a given key. No page contains rows from more than one key. However blobs (when not inlined on the row) will be placed in consecutive DB pages (up to their size). In addition to the blob and key pages the Virtuoso DB will hold a number of pages containing internal data. So the sum of the pages occupied by the key rows and the blobs is leas then the amount of occupied pages (as reported by the status() BIF).
To provide detailed information about the space consumption of each key there's a system view:
DB.DBA.SYS_INDEX_SPACE_STATS ISS_KEY_TABLE varchar -- name of the table ISS_KEY_NAME varchar -- name of the key ISS_KEY_ID integer -- id of the key (corresponding to KEY_ID from DB.DBA.SYS_KEYS) ISS_NROWS integer -- number of rows in the table ISS_ROW_BYTES integer -- sum of the byte lengths of all the rows in the table ISS_BLOB_PAGES integer -- sum of the blob pages occupied by the outline blobs on all the rows of the table ISS_ROW_PAGES integer -- sum of all the db pages containing rows of this key ISS_PAGES integer -- = ISS_BLOB_PAGES + ISS_ROW_PAGES (for convenience).
Each select on that view causes the server to go over all the db pages in the db file (similarly to how the crash dump operates) and collect the statistics above. The pages are traversed 1 time per select, but still on large database files that may take some time.
Bad design and implementation of transactions affects applications in the following ways:
The following rules should be observed when writing transactions:
The isolation level is set in Virtuoso/PL with the
set isolation := level;
statement, where level is one of 'serializable', 'repeatable', 'committed', 'uncommitted'. Example :
set isolation = 'serializable';
The standard SQL syntax is also supported :
SET TRANSACTION ISOLATION LEVEL <isolation_level> isolation level : READ UNCOMMITTED | READ COMMITTED | REPEATABLE READ | SERIALIZABLE
Example :
SET TRANSACTION ISOLATION LEVEL READ COMMITTED
The effect is for the rest of the procedure and any procedure called from this procedure. The effect stops when the procedure having executed the set isolation statement returns.
The text for a deadlock handler is
declare retry_count int; retry_count := 0; retry: { declare exit handler for sqlstate '40001' { rollback work; ... delay (rnd (2.5)); --- if 2.5 seconds is the expected duration of the transaction. ... retry_count := retry_count + 1; if (retry_count > 5) signal ("xxxxx", "Too many deadlock retries in xxxxx."); goto retry; } -- do the operations. The actual working code here. commit work; }
An exclusive read is done with
select s_quantity from stock where s_i_id = 111 for update;
For the Virtuoso ODBC driver the isolation is set by :
rc = SQLSetConnectOption (hdbc, SQL_TXN_ISOLATION, SQL_TXN_READ_COMMITTED);
rc = SQLSetConnectAttr (hdbc, SQL_TXN_ISOLATION, SQL_TXN_READ_COMMITTED, NULL);
Constants are : SQL_TXN_READ_UNCOMMITTED, SQL_TXN_READ_COMMITTED, SQL_TXN_REPEATABLE_READ, SQL_TXN_SERIALIZABLE
connection string element : You may specify the default transaction isolation level for a given connection in it's connect string (passed to the SQLDriverConnect () ODBC API). Example :
SQLDriverConnect (hdbc, hwnd, "DSN=MyDSN;IsolationLevel=Repeatable Read;UID=dba;PWD=dbapwd", SQL_NTS, NULL, 0, NULL, SQL_DRIVER_NOPROMPT).
The possible options for the connection string are : "Read Uncommitted", "Read Committed", "Repeatable Read" and "Serializable"
In the Virtuoso JDBC driver the isolation is set by the java.sql.Connection.setTransactionIsolation() JDBC API.
conn.setTransactionIsolation (java.sql.Connection.TRANSACTION_SERIALIZABLE)
The constants are described in the Java Docs
In the VirtuosoClient.NET provider the isolation is set by the System.Data.IDbConnection.BeginTransaction Method (IsolationLevel) function.
System.Data.IDBTransaction trx = conn.BeginTransaction (System.Data.IsolationLevel.ReadCommitted)
The constants are described here
Metrics are presented at the server and the table level.
The first metric to check is the output of status ('');
The paragraph titled transaction status contains the following:
The system view db.dba.sys_l_stat is used for locating bottlenecks.
The columns are:
All counts and times are cumulative from server startup.
The interpretation is as follows:
If deadlocks is high in relation to waits or locks, i.e. over 5%, there are many deadlocks and the transaction profiles may have to be adjusted. The table where deadlocks is incremented is the table where the deadlock was detected but the deadlock may involve any number of tables. So, if A and B are locked in the order A, B half of the time and B, a the rest of the time, then the deadlocks of the tables of A and B will be about the same, half of the deadlocks being detected when locking A, the other half when locking B.
If waits is high in relation to locks, for example 20%, then there is probably needless contention. Things are kept locked needlessly. Use read committed or make shorter transactions or lock items with the less contention first.
Because transaction duration varies, the place with the highest count of waits is not necessarily the place with the heaviest contention if the waits are short. Use wait_msecs in addition to waits for determining where the waiting takes place.
To get a general picture, use the Conductor's Statistics page or simply do
select top 5 * from sys_l_statt order by wait_msecs desc;
to get a quick view of where time is spent. You can also sort by waits desc or locks desc.
It is possible to get bad locking behavior if the SQL compiler decides to make linear scans of tables or indices and the isolation is greater than read committed. The presence of a linear scan on an index with locking is often indicated by having a large number of lock escalations. If lock_esc is near locks then a large part of the activity is likely sequential reads.
The general remedy is to do any long report type transactions as read committed unless there are necessary reasons to do otherwise.
To see how a specific query is compiled, one can use the explain () function. To change how a query is compiled, one can use the table option or option SQL clauses.
Deadlocks and contention do not occur uniformly across time. The occurrences will sharply increase after a certain application dependent load threshold is exceeded.
Also, deadlocks will occur in batches. Several transactions will first wait for each other and then retry maybe several times, maybe only one succeeding at every round. In such worst cases, there will be many more deadlocks than successfully completed transactions. Optimize locking order and make the transactions smaller.
Looking at how counts change, specially if they change in bursts is useful.
The most detailed picture of a system's behavior, including deadlocks nd other exceptions cn be obtained with profiling. If the application is in C, Java or some other compiled language, one can use the language's test coverage facility to see execution counts for various branches of the code.
For client applications, using the Virtuoso trace() function can be useful for seeing which statements signal errors. Also the profiling report produced by prof_enable () can give useful hints on execution times and error frequencies. See Profiling and prof_enable ().
For PL applications, Virtuoso provides a profiler and test coverage facility. This is activated by setting PLDebug = 2 in the Parameters section of the ini file and starting the server. The functions cov_store () and cov_report are used for obtaining a report of the code execution profile thus far. See the documentation on "Branch Coverage" for this. The execution counts of the lines for exception handler code will show the frequency of exceptions. If in a linear block of code a specific line has an execution count lower than that of the line above it, then this means that the line with the higher count has signalled as many exceptions as the difference of the higher and lower count indicates.
The times indicated in the flat and call graph profile reports for PL procedures are totals across all threads and include time spent waiting for locks. Thus a procedure that consumes almost no CPU can appear high on the list if it waits for locks, specially if this is so on multiple threads concurrently. The times are real times measured on threads separately and can thus exceed any elapsed real tiime.
Single stepping is not generally useful for debugging locking since locking issues are timing sensitive.
Starting with version 6, Virtuoso keeps track of the count of basic database operations performed on behalf of each connected client. The resource use statistics are incremented as work on the connection proceeds. The db_activity () function can be called to return the accumulated operation counts and to optionally reset these.
The db_activity built-in function has one optional argument. The possible values are:
The human readable string is of the form:
22.56MR rnd 1.102GR seq 10P disk 1.341GB / 102.7K messages
The postfixes K, M, G, T mean 10^3 to 10^15, except when applied to bytes, where these mean consecutive powers of 1024.
The numbers, left to right are the count of random row lookups, sequential row lookups, disk page reads, cluster inter-node traffic in bytes and cluster inter-node message count as an integer number of messages. If the configuration is a single server, the two last are 0.
The random and sequential lookup counts are incremented regardless of whether the row was found or not or whether it matched search conditions.
If the information is retrieved as an array, the array contains integer numbers representing these plus some more metrics.
Index - Meaning
If the thread calling db_activity is a web server thread, the totals are automatically reset when beginning the processing of the current web request.
Previous
Contents of Administration |
Chapter Contents |
Next
HTML based Administration Console (Conductor) Guide |