Managing spoolers

This chapter provides an overview of the concepts that are related to EnginFrame spoolers and their management.

A spooler is a dedicated data container that EnginFrame creates to host files that users provide (for example, input files uploaded using a web browser) or files that are generated by the services (for example, output or temporary files).

Every time the service is run (unless explicitly configured) causes EnginFrame to create a new spooler with appropriate user permissions that allow services to read from and write to spooler's directory.

Under EF_SPOOLER_DIR, EnginFrame creates a directory for each user the first time that the system is accessed. These directories are named using the user's names. Under each <username> directory, EnginFrame creates its spoolers, one for each time the service is run.

The following image provides a visual overview of EnginFrame's spooler directory structure.

Spoolers directory structure

In the preceding image, Jane, Bob, Lucy each have their own spooler directories. tmp1.ef, tmp2.ef, tmp3.ef, tmp4.ef, tmp5.ef are spooler directory names that EnginFrame created dynamically.

Spoolers requirements

If agents that use the spoolers run on hosts other than the server hosts, the spoolers must reside on a shared file system. This shared file system must be mounted from the EnginFrame Server's host. Both the agents and server must be able to read and write to the shared file system. With theEnginFrame's mapping mechanism, these file systems can be mounted with different paths on server and agent hosts.

Note

If you're using multiple agents, the same mount point path must be used for all of them.

EF_SPOOLER_DIR directory must be owned by the user that's running EnginFrame Server (for example, efnobody). The same userefnobody must have read and write permissions. This is because EnginFrame Server initially creates spoolers for users. All other users must also have read and run permissions for the spooler root directory. In other words, the spooler root directory must have the following ownerships and permissions: rwx r-x r-x efnobody:efnogroup where efnogroup is efnobody's primary group.

The spoolers area must be placed on a file system where the agent's owner has complete read and write privileges. Depending on who runs agent daemon, this can be achieved by one of the following tactics:

• Avoiding root squashing on NFS server when agent owner is root.

• Using same user running server when agent owner is a normal user.

Note

Distinct EnginFrame deployments can't share the same spooler area.

Spooler security permissions

EnginFrame Server starts with a user file creation mask, such as umask, set to 022. This means the following happens:

• Files are created with 644 rw-r--r-- permissions.

• Directories are created with 755 rwxr-xr-x permissions.

This assures EnginFrame Server user can both read and write files in spoolers. This might lead to a weak security environment because every user can read files from spoolers that are owned by other user accounts. For this reason, EnginFrame Agent changes spooler permissions to 750 rwxr-x--- just before the service is run. In other words, the final spooler permissions are read, write, and run for the spooler's owner and read and run for EnginFrame Server's owner. This occurs because user efnobody with group efnogroup creates the spooler. EnginFrame Agent changes spooler's owner by running chown <username> <spooler> and spooler ownership becomes <username>:efnogroup. This allows the spooler's owner to perform whatever action is needed inside spooler and that only efnogroup members are allowed to read data inside spoolers. A security best practice is to have only efnobody as efnogroup group's member.

Configuring EnginFrame spoolers

This section describes how to customize basic spoolers settings concerning directory paths and file download aspects.

Configuring spoolers default root directory

EF_SPOOLER_DIR directory is chosen during installation. All spoolers are created by EnginFrame under this directory. The default value is $EF_TOP/spoolers.

Spoolers root directory can be changed by editing EF_SPOOLERDIR parameter inside $EF_TOP/conf/enginframe.conf. The startup parameter name EF_SPOOLERDIR is slightly different from the property EF_SPOOLER_DIR that's set and used inside EnginFrame.

Example Change the spooler default root directory

EF_SPOOLERDIR=/mnt/scratch/spoolers

After this change, all spoolers are created under /mnt/scratch/spoolers directory.

Note

Changes to this parameter require an EnginFrame Server restart.

Note

Spooler location is defined at creation time, so changing EF_SPOOLERDIR doesn't affect an existing spooler's location. EnginFrame Agent retrieves old spoolers from the old location and new spoolers from the new location.

Refer to Spoolers requirements when setting up a new spooler root directory.

Download files from spoolers

File are downloaded from spoolers on the user's behalf through the server and agent. EnginFrame Server receives a download request and forwards it to the agent that accesses the file. The agent posts the file back to server using HTTP or HTTPS.

This process implies EnginFrame Agent is able to connect back to server using HTTP or HTTPS for sending data. As a result, make sure that you are careful with how you configure your network and firewall.

Note

If EnginFrame Server is configured to accept requests through HTTP over SSL - HTTPS - protocol, then refer to the Configuring HTTPS topic to configure the server and agent.

Consider setting a mime-type for downloaded files to let browsers identify the file's type. You can customize and configure the mime-type that EnginFrame uses when downloading files. Managing internet media types provides instructions on how to set up mime-type configurations.

Configure download URL on agent

The download process highlights how an agent has to connect back to the server for sending downloaded data. Usually the server HTTP endpoint, that is, the host and port that the agent connects to, is automatically detected from server's request.

There might be network configurations or architectural scenarios that require specific configurations. For example, web access might be configured with an HTTP server in front of EnginFrame Server. In this case, the agent must use a different host and port to connect to the server. To enable this, you must explicitly configure the complete URL that the agent must use to connect to the EnginFrame Server.

The ef.download.server.url parameter inside $EF_TOP/conf/enginframe/agent.conf (or $EF_TOP/<VERSION>/enginframe/conf/agent.conf) sets the URL agent uses to connect to server. It's defined as follows:

ef.download.server.url=http[s]://<host>:<port>/<web-context>/download

Replace the following elements with your specific details:

• The host and port value identifies the EnginFrame Server network endpoint.

• The web-context value is the root context that you chose when installing EnginFrame. By default, it's enginframe.

The following is modified with example values for illustration purposes.

ef.download.server.url=http://localhost:8080/enginframe/download

Configure streaming download timeout

Streaming downloads have a timeout setting. If the file being streamed doesn't change during this set interval of time, EnginFrame interrupts its stream that's being sent to a client. By default, this value is set to 300 seconds.

This value is specified in $EF_TOP/conf/enginframe/server.conf (or $EF_TOP/<VERSION>/enginframe/conf/server.conf). You change this value by editing ef.download.stream.inactivity.timeout property inside $EF_TOP/conf/enginframe/server.conf. The value is expressed in seconds.

In the following example, the timeout value is set to 600 seconds (10 minutes).

ef.download.stream.inactivity.timeout=600

Configure streaming download sleep time

The file streaming download feature of EnginFrame works using a pull model. The server periodically queries an agent for available data.

You can set the specific interval of time between two subsequent checks. By default, it's set to 5 seconds. A lower interval time might improve the user experience but it might also increase system load.

This value is specified in $EF_TOP/conf/enginframe/server.conf (or $EF_TOP/<VERSION>/enginframe/conf/server.conf). You change this value by editing ef.download.stream.sleep.time parameter inside $EF_TOP/conf/enginframe/server.conf. The value is expressed in seconds.

For example, with ef.download.stream.sleep.time=20, the sleep time is set at 20 seconds.

Spooler life cycle

This section outlines EnginFrame's spooler lifecycle. Besides outlining spooler's lifecycle, each sub-section describes common customizations that you can apply to EnginFrame Portal.

Overview

Spoolers are created for each service submission. More specifically, spoolers are created for all those services whose spooler definition has a TTL different from -1.

Spoolers are initially created by EnginFrame Server with a defined time-to-live. Together with spooler directory the server also creates an entry in EnginFrame's spooler database called repository. The repository is file-system based and each entry is a file that contains all information that's necessary to recreate a spooler.

• The owner

• The physical location path on both server and agent

• The display name

• And other properties

The server also has the responsibility to save user's files into a spooler before contacting an agent for running a service. After the spooler setup tasks have been accomplished, the server contacts an agent to run the service. The agent first changes the spooler's and its contents ownership and then uses this spooler as the working directory for the service it runs.

EnginFrame removes a spooler when its life-time expires, deleting spooler's directory with its content and its repository entry. An EnginFrame thread called reaper periodically checks if there are expired spoolers ready to be removed.

Change repository location

The default EnginFrame repository path is $EF_TOP/repository. With EnginFrame, you can change the location of where repository entries are stored.

For example, you might want to save repository entries on a high speed and reliable file-system or on an area that has "dynamic" file-system paths (for example, a directory with contents that change often) to conform to your company's policies.

The repository location is configured by EF_REPOSITORYDIR parameter inside the $EF_TOP/conf/enginframe.conf file. It specifies an absolute path in the file system. You can use other variables that are defined in enginframe.conf for path definition. For example, EF_REPOSITORYDIR=$EF_TOP/repository defines repository location using $EF_TOP variable. EF_REPOSITORYDIR=/mnt/nas/ef-repository sets an absolute path for repository directory.

Note

If you make changes to EF_REPOSITORYDIR, you must restart EnginFrame Server.

Configure reaper sleep time

The EnginFrame reaper is a thread that periodically wakes up to check if there are expired spoolers in the system needing cleanup.

You can configure reaper's thread sleep interval to specify how frequently this check is performed.

This value is specified in $EF_TOP/conf/enginframe/server.conf (or $EF_TOP/<VERSION>/enginframe/conf/server.conf). You change this value by editing the ef.reaper.sleep.time property inside $EF_TOP/conf/enginframe/server.conf file. The value is expressed in minutes. By default, it's set to 30 minutes.

In this example, ef.reaper.sleep.time=60, the sleep time is set to 60 minutes (1 hour) between each thread's reap.

Spoolers removal: Dead spoolers

If for any reason spooler cleanup fails, spooler's directory is renamed with the DEAD_ prefix added in front of its original name.

For example, if EnginFrame can't remove spooler tmp32062.ef, it's renamed to DEAD_tmp32062.ef. You can remove dead spoolers from your system safely.

If you set up deadspooler logging target after a dead spooler was created, the event is logged to the $EF_TOP/logs/DEAD_spoolers.{agent|server}.log depending on whether the server or agent had an error.