AVL Release Notes


New Features

Universal Lab Launcher

This feature allows for easy embedding of AVL labs into any web page.  For more information check out the documentation.


Bug Fixes

Correct Launching Error Results

In some cases, attempting to launch a non-existent project, or trying to launch a project without a user email would give an incorrect error response.  This is now giving the correct error response.


New Features

Deployment Event Logs

Deployment Event logs are written and available to download from: <yoursite>/reporting/raw_history/csv

The events recorded are:


Bug Fixes

Allow Lab Admins to change port settings of existing Labs

Previously the admin user could not update the port settings once a lab had been launched, which caused the Appsembler team to have to update the settings with a workaround. Now if there are no running labs tied to these image settings, the ports can be modified.

Clean up containers after failed launch

This fix consists of two parts. The first part fixes the issue where labs were not always cleaned up well after they have failed to launch. The second part is to run a script to periodically clean up these containers to handle any failures from before the fix was applied. The script continuing to run is a precautionary measure in the case of other scenarios where we don't clean up properly, though as yet we don’t know of any bugs in lab cleanup that have not been resolved.


Handle failed lab launches more gracefully

In some cases, the response from the lab launch was not handled well, resulting in the learner seeing an opaque error message or no message at all. Now the learner will always see a message and the message will be informative.


This version was not released.


New Features

No significant new features were added in this release

Bug Fixes

No bug fixes were included in this release


Fix and expand event logging

Throughout the lab lifecycle, we log events, and we have added two new event types: when a deployment fails and when a deployment is ready. Along with the existing events, for deployment started and deployment deleted, this can help us and you more fully to understand why a deployment was removed—simply because it came to the end of life, or because something went wrong in the launch process.

Handle timeouts better

Occasionally, a request for a lab will timeout in our application (rather than by us hitting the threshold for ping_retry). When this happens, we now respond more gracefully to the learner and log more information for us to debug and improve.


New Features

No significant new features were added in this release

Bug Fixes

Use freed reserved host ports

The fix in 5.3.0 was incomplete and in certain cases where containers allocate a lot of ports, the ports were not reused correctly, this is resolved with this patch.


No other improvements are included with this release.


New Features

No significant new features were added in this release

Bug Fixes

Ensure the User Containers tab of the AVL dashboard is always available.

Instead of the User Containers tab displaying an error, it will show the unhealthy labs, but grey them out.
Note: Recently-launched labs may display as “unhealthy” if the dashboard was loaded as the lab was starting before it has fully launched.


Improve cleanup up unhealthy labs

As part of the work above, changes were made that unhealthy labs get cleaned up correctly.


New Features

No significant new features were added in this release

Bug Fixes

Use freed reserved host ports

Recently we’ve run into an issue with “reserved ports” in AVL.  Each lab needs a number of ports open on the host machines so that the learner can access them.  The limit was 20,000 total ports, over the lifespan of the cluster, and some customers hit this limit recently.  With v5.3, these ports are now reused, which will also eliminate the symptoms of the port database table filling up (excessive logging and disk issues).

Add correct class name error

When an admin user tried to change the name of an event's class, the error wasn't displaying properly.

Fix the clickable area of the "Add port..." button in the image settings widget

The button for adding a port was sometimes obscured.

Correctly display error when changing Event Class name

When an admin user tried to change the name of an event's class, the error wasn't displaying properly.


Add AVL_DEPLOY_ID to container env variables

In a lab, access the lab's deploy_id via the AVL_DEPLOY_ID env var.

Update NGINX buffer config

A limitation in the web proxy was identified with one customer, that affected requests with a large header size not being handled correctly. Note this is an optional deployment on an AVL cluster and will be enabled as required.

Notify if deleting an Image will also remove an Admin Container

In some cases, deleting an image would delete an admin container without first notifying the admin. We now stop before deleting to confirm


We updated a key package that handles queuing lab launching and cleanup tasks.  This will both give us further insight into why labs don’t always launch and improve the effectiveness of the launching process. 


This release contained internal improvements only.


New Features

Display paired container logs

Added paired container's logs tab for user containers.

Improve Container Runtime Selection

Containers can now have their runtime set with units of a second, rather than days.  The minimum run time is 1 hour and the maximum is 60 days.

Add Memory Allocation to reporting

The raw_history/csv output was not providing the full info for the images. With this change, we are able to see the RAM allocation for both, primary and paired images, which would allow us to estimate capacity at all times.

Bug Fixes

Properly set X-Forwarded-Proto header for container requests

HTTPS requests between students and labs were broken under certain circumstances. The current flow goes:

  1. A student initiates an HTTPS connection by clicking on a URL
  2. The request hits Nginx and terminates the TLS connection
  3. The resulting HTTP connection is proxied to the indicated Student Container and port
  4. The response is returned to the student's browser

In some cases, the application running on the student container would then send absolute links in the page content, including the protocol that it received the request on, incorrectly as HTTP.  Updating the X-Forwarded-Proto header fixes this problem and ensures the protocol for future connections matches that of the connection used for the initial page load.

Make sure Event slugs are unique

Fixed a bug that allowed non-unique URL slugs to be generated from similarly named Events


Update default image setting to use https

Update the default HTML widget code in image settings to use https rather than http in the container access URL.  This only affects newly created image settings.


New Features

Added token authentication to Projects endpoint

This allows for programmatic access to the list of AVL projects in a cluster

Serialized http_content field content in its serializer

Ensure that the API for launching containers returns the HTML snippet correctly in the JSON response

Added start and end dates to query string for /reporting/raw_history/csv endpoint

This allows for selecting a range of data for the reporting API endpoint to return

Bug Fixes

Made the Image Settings icon visible when the name is too long

In cases where an Image Settings had a long name, the button to edit the settings was not displayed in the AVL interface, this corrects that bug

Fixed environment variables in paired containers

This fixes a regression in v5.0.0 where the AVL environment variables were incorrectly set in Paired containers. This did not affect Primary containers.

Fixed footer logo on event pages

This fixes a bug on Event Student Dashboards where the Appsembler Logo overlapped the page content and prevented clicking the lab links.


Implement correct deployment API response codes

The deployment API now returns more accurate HTTP response codes for success states, error states remain unchanged

Reload Nginx after renewing Lets Encrypt SSL certificate

Ensure that when the SSL certificate on an AVL cluster is refreshed, Nginx is reloaded to correctly serve the new certificate.


New Features

Modify project's remaining deployments

We manage a project's remaining deployments differently than pre-v5, now that we have container cleanup on expiration. The number of remaining deployments is now calculated from the number of active deployments (those that are "running" or "stopped") rather than being calculated by summing the total number of historical deployments. So, if a project is configured to allow ten deployments, then launches five, then one expires, the project will have six (not five) deployments remaining.
This new behaviour was requested by several of our customers in the past. But please take note if you’ve been using remaining deployments as a means to cap the total number of deployments ever created – this behaviour has changed.

Bug Fixes

Fix Certbot across customer deployments

We made fixes to improve and standardize our use of Certbot across customer deployments. Certbot is part of the Let’s Encrypt ecosystem, and we use it to get and refresh SSL certificates for AVL.
This fix increases system stability and improves maintenance, but is informational – there should be no admin- or learner-facing change.

Fix race condition around new container creation

Fixed a race condition that allowed two containers to be spun up for the same Learner and Project. This caused problems in other parts of the codebase that only expect a single LearnerDeployment to be returned for a given (Learner, Project) pair.
This was one of the key bug fixes that we identified in v5.0.0 and prevented us from rolling out v5 more broadly.

Include all timezones in the event creation modal

A few users had noted missing timezones in the Event creation popup. 
We were manually limiting the time zones that were displayed in that dropdown. We have now restored the missing timezones.

Restart stopped container with a call to /isc/newdeploy

We fixed a bug that prevented stopped containers from being restarted on subsequent calls to the /isc/newdeploy endpoint. The stopped container is now restarted, and then the container’s information is returned.


Increased timeout for container start

For some images, the containers created from the image take a long time to spin up. In these cases, AVL was hitting a timeout before the request was completed.
This PR increases the timeout on the container creation request from 60 seconds to 90 seconds. The real solution lies in the software itself – speedup time needs to decrease. The lab author needs to speed up the time it takes the container to start. However, this should give extra time.

Performance improvements for learner_deployments endpoint

For AVL instances with a large number of learner deployments, accessing the learner deployments endpoint could lead to long load times, and repeated calls could overwhelm the main node. The fix involved adding date, email, event_name, and event_class_name filters to the /isc/dashboard/learner_deploymentendpoint. 


v5 is a refactor of our code, which will allow us to more quickly add new features and fix bugs.

New Features

Option to auto-delete labs at expiration

When the lab hits the "expiration date," it is now possible to take the (more reasonable) path: the lab is deleted rather than merely stopped. This is an optional feature. Unless we've talked with you, your labs will not be deleted, so plan to continue deleting manually. Let us know if you want your labs to be automatically deleted. (Note: this is a site-wide flag, so every lab will be deleted at the expiration date. Soon we hope to allow a more granular option, e.g., auto-deleting per-project.)

Container name and URL changes

During the deployment of v5, we will change the name of each lab. All labs will now use learner___<deployID> instead of user___<deployID>. The URL of each lab will also change, so you may encounter a learner here and there with stale links (for example, if a learner has saved the link as a bookmark). If you encounter an issue with a dead link, please ask these learners to return to the course page and click the (re)launch button for the lab. That button will return the new URL (but of course, it will be the same lab and lab content).
This change in URLs comes about because we are migrating all labs in the database. For a long time, we've had two types of labs in the database: one for live events and another for project / on-demand containers. Combining these into one table significantly reduces the complexity of our application and, as noted above, will allow us to complete features and bug fixes more efficiently and effectively. URLs work slightly differently with the learner___ structure, to which we're moving, so each migrated lab needs to align with this format. Hence, each lab URL must change.

Failed containers are deleted

If a container fails to launch, it will be automatically deleted. We consider a container "failed" when any of the steps do not work, including the final step where we check to see if your processes are running in the container. We realize that this may not be ideal in some cases; e.g., if you need to check the logs from the failed container or if you take other actions even when the container fails to launch. In v5.0.4, the next release after this one, we intend to improve this. However, if your processes are broken by not having failed containers in the admin UI, you may want to wait on getting v5. Please let us know if this is the case for you.