Arches 7.5.0 Release Notes
--------------------------

### Major enhancements
- Upgrade Django (and related packages) to 4.2 #9866
- Support Python 3.11 #10200
- Add Tile Excel format importer to bulk data manager #9978
- Add Branch and tile Excel exporters to bulk data manager #9978
- Improve bulk data manager editors #10156 #10268 #9952
- Save workflow progress to database to persist history and track incomplete workflows #9959, #10154
- Add image mode to file-list datatype to save alt text and other image metadata #9934
- Support Arches Applications #9426
- Improve Accessibility #10265

### Performance Improvments
- Add edit log index for bulk editor #10098
- Save one query per subsequent resource when recalculating descriptors #10312 #10316
- Improve performance when saving resource_x_resource records #9747
- Improve bulk load performance #9714

### Additional improvements and bug fixes

- Add map config hook for map widget customization #10354
- Add generic search thumbnail retrieval #10266
- Show large thumbnail preview when hovering on search result thumbnail #10317
- Add code to support thumbnail generation (currently noop) #10320
- Avoid trying to update the publication of an unpublished graph in graph publish command #10410
- Demote no-ontologies UI error to a logged warning #10385
- Fix string widget placeholder #10380
- Fix bulk import django_storage path for remote file storage #10377
- Fix map header minzoom #10374
- Update language synchronizer to serialize graph proxy model #10373
- Fix missing role and label for notification dismiss button #10345
- Fix missing image alt text for accessibility #10344
- Add system check for using dummy cache in production #10340
- Fix the missing canvas label #10337
- Remove `mock` from dev dependencies #10325
- Add contenttypes dependency to initial migration #10303
- Fix function registration #10301 & #10286
- Make primary descriptor fn aware of which description it's calculating #10300
- Update models pattern && removes plugins directory #10290
- Fix resource.load_tiles method to only filter if given perm #10275
- Do not require hostname in resource instance search string #10272
- Add missing loadid arg #10263
- Add uuid field to iiif-manifest #10261
- Superusers can finish workflows of other users #10256
- Check if function is new in save logic #10253
- Add plugins api #10241
- Fix webpack symlink mismatch #10236
- Make file-widget easier to inherit from #10233
- Removes full-screen overlay for frontend errors when running webpack development server #10225
- Fix race condition in resource-instance-select.js #10211
- Support HTML in manifest attribution field #10208 #10209
- Add map-filter auto zoom behaviour setting #10207
- Surface more meaningful error for missing graph publications #10189 #10203
- Add new configuration to disable automatic map extent change behaviour in map filter #10195
- Adds permission check for related resources #10191
- Add `defaultValue` to file-widget default config #10181
- Properly assign provisional flag during index, #10179
- Add graph slug as css class #10177
- Place JSON configs for card components etc in correct place #10164
- Fix race condition in concept select widget #10163
- Remove duplicative spinner in `.search-result-details` #10161
- Fix saving unsaved data in finish workflow action #10158
- Assign value to missing parameter in bulk editor #10156
- Creates concat str from node names on card in advanced search facets #10148
- Fix language selection in text widgets #10122 #10140
- Fix exponential files upload bug #10133
- Fix dirty state indication in the date picker widget #10126
- Add the capability of replacing the whole word only in bulk editor #10124
- Assign view_plugin permission to Resource Reviewers #10064 #10118
- Adds update flag to graph publish command #9625 #10116
- Fix TypeError when deleting all files from tile #10115
- Add tile deletion to bulk data manager #10101
- Enforce that skos file hasTopConcept #10100
- Move workflow history from local storage to database #10096
- Update fix for missing graph ids in resource x resource table #10092
- Adds missing data in the edit log following bulk edit/import #10073
- Add a button to copy node UUIDs to clipboard in graph designer tree #10071
- Limit the number of bulk edits #10066
- Add link to the list of bulk edits/imports in edit history page #10056
- Fix canvas delete function #10053
- Fixes issue where setting the arches version stage to "alpha" or "rc" stops package.json being created correctly #10051
- Allows naming of exported files from bulk_data_manager #10049
- Adds option for users to export concepts as UUIDS when using bulk tile exporter #10047
- Let errors bubble up in `ManifestManagerView` #10046
- Add a "View Report" button to resource editor #10045
- Add node alias and widget name as class to HTML elements #10034
- Add the count of resources/tiles imported in bulk loader #10032
- Add the file name in single csv importer summary #10021
- Update staging data for bulk edit #10020
- Update the logic to stage data for bulk edit #10019
- Remove workflow hardcoding in workflow banner #10015
- Allow configuration of uploadedfiles directory name #10013
- Fix bulk edit operations when data contains escape chars #10008
- Improve manifest selection verbiage in manifest manager #10004
- Enforce ordering in bulk disambiguated endpoint #9995
- Update webpack config #9993
- Allow importing empty worksheets in bulk loader #9991
- Add search debounce of 500ms AB#59153 #9946
- Add workflow step to check for missing migrations #9921
- Remove nose, subclass Django's test runner #9887
- Add support for Elastic Cloud #9860
- Add support for Elasticsearch API key authentication #9859
- Fix for graphs with forward slash ("/") in name when using create_package  #9778
- Support provisional edits in resource json #9717
- Fix for allowing double quotation marks in i18n inputs #9712
- Fix errors when creating provisional edits #9704
- Prevent search export of restricted tiles #10425
- Handle deletion verify at card component level #9096

### Dependency changes
```
Python:
    Upgraded:
        Django 3.2.20 > 4.2.8
        django-celery-results 2.4.0 > 2.5.1
        django-compressor 3.1 > 4.4
        django-cors-headers 3.1.1 > 4.2.0
        django-guardian 2.3.0 > 2.4.0
        django-recaptcha 2.0.6 > 3.0.0
        django-oauth-toolkit 1.2.0 > 1.7.0
        django-webpack-loader 1.5.0 > 2.0.1
        celery 5.2.7 > 5.3.6
        psycopg2 2.9.6 > 2.9.9

    Added:
        django-revproxy==0.12.0

    Removed:
        arches-django-revproxy

        (dev dependencies):
            webtest
            django-webtest
            django-nose
            mock

JavaScript:
    Upgraded:

    Added:
        select-woo: jadu/selectWoo/tree/jadu-1.1.5
        vue: ^3.3.4

        (dev dependencies): 
            vue-loader: ^17.2.2,
            vue-template-compiler: ^2.7.14

```

### Breaking changes
- If your project has its own Django models with custom save methods, you may need to modify them to ensure your fields are properly set using `update_fields`. See the [Django release notes](https://docs.djangoproject.com/en/4.2/releases/4.2/#setting-update-fields-in-model-save-may-now-be-required) for details.
- The iiif_manifest table now has a globalid column (uuid). Imports to this table will need a uuid value for this field.
- Descriptors will not always be calculated and saved on index. Custom import scripts will need to call `resource.save_descriptors()` before `resource.index()` to ensure descriptors are updated. 
- The `calculate_descriptors` method on the Resource proxy model has been renamed `save_descriptors`. Custom import scripts may need to reflect this change.
- The `select2-query` knockout binding now uses the `selectWoo` JS library in order to provide better accessibility than `select2`. As of this change, any HTML element with a binding like `data-bind="select2Query: { ... }"` must be a `<select>` element. These elements should also have an aria-label, however the normal `aria-label` attribute will not work. To do this, use `attr: {'data-label': ...}` in the binding. 

### Upgrading Arches

1. You must be upgraded to at least version 7.4.0 before proceeding. If you are on an earlier version, please refer to the upgrade process in the [Version 7.4.0 release notes](https://github.com/archesproject/arches/blob/dev/7.4.x/releases/7.4.0.md)

2. Be sure to backup your database before proceeding.

3. Upgrade to Arches 7.5.0
    ```
    pip install --upgrade arches==7.5.0
    ```

4. Uninstall mooted dev dependencies:
    ```
    pip uninstall webtest
    pip uninstall django-webtest
    pip uninstall django-nose
    pip uninstall mock
    ```

### Upgrading an Arches project

1. In urls.py, replace each `django.conf.urls.url()` call with `django.urls.re_path()` if it uses a regular expression, or `django.urls.path()` if it does not.

2. Replace any calls to `django.utils.translation.ugettext()` with `django.utils.translation.gettext()`.

3. In settings.py (or settings_local.py, etc.), remove `USE_L10N`. If present, replace `DEFAULT_FILE_STORAGE` with `STORAGES["default"]["BACKEND"]` and/or `STATICFILES_STORAGE` with `STORAGES["staticfiles"]["BACKEND"]`. (An example configuration can be found in the [Django documentation](https://docs.djangoproject.com/en/4.2/ref/settings/#storages).)

4. Within your project, with your Python 3 virtual environment activated:
    ```
    python manage.py makemigrations  # commit the result
    python manage.py migrate
    ```

2. Reindex your database:
    ```
    python manage.py es reindex_database
    ```

3. If you have made customizations to files in your webpack directory, backup that directory as those files will be overwritten in the following steps. Read [this](https://github.com/archesproject/arches/blob/dev/7.5.x/arches/webpack/README.md) for more information.

4. If you have javascript dependencies listed in `webpack/webpack-node-modules-aliases.js`, move them to your package.json file and update the paths. For example, if you have dependencies in `webpack/webpack-node-modules-aliases.js` like this:
    ```
    module.exports = {
        PROJECT_NODE_MODULES_ALIASES: JSON.stringify({
            "plotly.js-dist": "Path.resolve(__dirname, `${APP_ROOT}/media/node_modules/plotly.js-dist`)",
            "Plotly": "Path.resolve(__dirname, `${APP_ROOT}/media/node_modules/plotly.js-dist`)"
        }),
    };
    ```
    They must be moved to an object called `nodeModulesPaths` in your project's package.json like this:
    ```
    "nodeModulesPaths": {
        "plotly.js-dist": "node_modules/plotly.js-dist",
        "Plotly": "node_modules/plotly.js-dist",
    },
    ```

5. Run `python manage.py updateproject`

6. Update your settings.py file:
    1. Add the ARCHES_APPLICATIONS SETTING:

        ```ARCHES_APPLICATIONS = ()``` 

    2. Replace:
        ```
        STATICFILES_DIRS = (
            os.path.join(APP_ROOT, "media", "build"),
            os.path.join(APP_ROOT, "media"),
        ) + STATICFILES_DIRS
        ```
        with:
        ```
        STATICFILES_DIRS =  build_staticfiles_dirs(
            root_dir=ROOT_DIR,
            app_root=APP_ROOT,
            arches_applications=ARCHES_APPLICATIONS,
        )
        ```

    3. Replace:
        ```
        TEMPLATES[0]['DIRS'].append(os.path.join(APP_ROOT, 'functions', 'templates'))
        TEMPLATES[0]['DIRS'].append(os.path.join(APP_ROOT, 'widgets', 'templates'))
        TEMPLATES[0]['DIRS'].insert(0, os.path.join(APP_ROOT, 'templates'))
        ```
        with:
        ```
        TEMPLATES = build_templates_config(
            root_dir=ROOT_DIR,
            debug=DEBUG,
            app_root=APP_ROOT,
            arches_applications=ARCHES_APPLICATIONS,
        )
        ```

    4. Replace:
        ```
        if __name__ == "__main__":
            print(
                json.dumps(
                    {
                        "ARCHES_NAMESPACE_FOR_DATA_EXPORT": ARCHES_NAMESPACE_FOR_DATA_EXPORT,
                        "STATIC_URL": STATIC_URL,
                        "ROOT_DIR": ROOT_DIR,
                        "APP_ROOT": APP_ROOT,
                        "WEBPACK_DEVELOPMENT_SERVER_PORT": WEBPACK_DEVELOPMENT_SERVER_PORT,
                    }
                )
            )
            sys.stdout.flush()
        ```
        with:
        ```
        if __name__ == "__main__":
            transmit_webpack_django_config(
                root_dir=ROOT_DIR,
                app_root=APP_ROOT,
                arches_applications=ARCHES_APPLICATIONS,
                public_server_address=PUBLIC_SERVER_ADDRESS,
                static_url=STATIC_URL,
                webpack_development_server_port=WEBPACK_DEVELOPMENT_SERVER_PORT,
            )
        ```

7. If your project is running on a port other than 8000, set the PUBLIC_SERVER_ADDRESS to that location, eg:
   
    ```PUBLIC_SERVER_ADDRESS = 'http://localhost:8009/'```

    **NOTE:** If you have modified `ARCHES_NAMESPACE_FOR_DATA_EXPORT` in your project's `settings.py` file, `PUBLIC_SERVER_ADDRESS` should be set to that value.


8. Update your project's JavaScript dependencies:

    1.  In the project's `package.json` file from `stable/7.4.0` to `stable/7.5.0`:
        ```    
        {
            "dependencies": {
                "arches": "archesproject/arches#stable/7.5.0",
            },
            "devDependencies": {
                "arches-dev-dependencies": "archesproject/arches-dev-dependencies#stable/7.5.0"
            }
        }
        ```

    2.  Ensure `nodeModulesPaths` has been added to your `package.json` file:
        ```    
        {
            "dependencies": {
                "arches": "archesproject/arches#stable/7.5.0",
            },
            "devDependencies": {
                "arches-dev-dependencies": "archesproject/arches-dev-dependencies#stable/7.5.0"
            },
            "nodeModulesPaths": {}
        }
        ```

    3. Remove your `media/node_modules` directory and `yarn.lock` file

    4. In your terminal navigate to the directory with your project's package.json file. Then run:

        yarn install

        
9. Start your application server in a separate terminal if it's not already running.

10. In the same terminal window where you ran `yarn` ( on the same level as `package.json` ), run `yarn start` or `yarn build_development`. This will generate your `media/build` directory.
    - If running your project in development:
     -  `yarn start` will build the frontend of the application and then start a webpack development server
      - `yarn build_development` will build a development bundle for the frontend assests of the application -- this should complete in less than 2 minutes
    - If running your project in production:
      - `yarn build_production` This builds a production bundle. **takes up to 2hrs depending on resources**
      - Alternatively you can `cd ..` up a directory and run `python manage.py build_production`. This will create a production bundle of frontend assessts and also call `collectstatic`.

11. If you are running Arches on Apache, be sure to run:

    ```
    collectstatic
    ```
    and restart your server
    ```
    sudo service apache2 reload
    ```