# Python Application Analysis

Python scanning supports binary packaged archives (`.whl`/`tar.gz`) and coordinates in the `requirements.txt` manifests from the [Python Package Index (PyPI)](https://pypi.org/). For the best results, we recommend first creating a Python virtual environment and resolving the dependencies using a pip install against the requirements file. This will bring only the dependencies needed by the project into the build while resolving any included dependency ranges in the requirements file.

Starting with [Sonatype IQ CLI 2.0](https://help.sonatype.com/en/sonatype-iq-cli.html "Sonatype IQ CLI"), Python `Pipfile.lock` files are supported, enabling accurate and efficient dependency identification directly from Pipenv projects. This feature eliminates the need for manual workarounds or complex CI configurations when analyzing Python projects with Pipenv. Note that if you are using the Jenkins plugin, you will need to upgrade it to benefit from this functionality, as the changes were made within the scanner.

## How to get the best results

To produce the best outcome the following is suggested:

- Create a `requirements.txt` file by using "pip freeze > requirements.txt".
- Using `requirements.txt`, download the binaries by executing "`pip download -r requirements.txt -d <output_dir>`" or "`pip wheel -r requirements.txt -w <output_dir>`".
- Run a scan on `<output_dir>`.

1. `cd project_folder`
2. `mkdir ~/py-envs`
3. `python3 -m venv ~/py-envs/project_folder`
4. `source ~/py-envs/project_folder/bin/activate`
5. `pip install -r requirements.txt`

Guidelines for the **requirements.txt** file:

- Use `pip freeze` to create the requirements file.
- Lifecycle scanners will only use the manifest named `requirements.txt` while ignoring other variants.
- Avoid including variable ranges for dependencies as these will be ignored by the Lifecycle scanner.
- The `requirements.txt` must use the == operator and version without wildcards.
- Additional flags should be added to `requirements.txt` files to scope to the target os/arch as found in the [environment markers](https://peps.python.org/pep-0345/#environment-markers) of the Python documentation.

**.whl files**: may be matched to multiple environmental Python packages. These will show as duplicates in the Lifecycle scan report.

## Evaluation: Source code and manifest analysis

The Python coordinate-based matching feature provides the ability to scan and evaluate Python dependencies found in Python manifest files without running the pip install first. Files named `requirements.txt` (generated using a pip command) and `poetry.lock` (Poetry lock files) will be analyzed.

### Converting from other formats

#### setup.py

A `setup.py` file can be used to generate a `requirements.txt` file by first installing its packages (e.g. via `pip install .`), ideally to a virtual environment, and then running:

- `pip freeze > requirements.txt` for Python 2 or
- `pip3 freeze > requirements.txt` for Python 3

## What do we parse from the file?

### requirements.txt

Requirements using the "==" operator and version without wildcards will be considered. One requirement could be matched to multiple distributions of the same Python package. Using the sys-platform marker makes the dependency more specific. For example:

| altgraph==0.10.2<br>pywin32 ==1.0 ; sys_platform == 'win32' |
| --- |

### poetry.lock

Refer to the sample _poetry.lock_ excerpt below:

Dependencies with _name_ and exact _version_ in the [[package]] section are required and evaluated. For example:

- Name: six
- Version: 1.16.0

Dependencies with _extension_ and _qualifier_ in the _[metadata.files]_ are required and evaluated. For example:

- Extension: whl
- Qualifier: py2.py3-none-any

Package dependencies with _name_ and exact _version_ in the _[package.dependencies]_ section are evaluated. For example:

- Name: colorama
- Version: 0.4.4

#### Sample poetry.lock

```
[[package]]
name = "six"
version = "1.16.0"
description = "Python 2 and 3 compatibility utilities"
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2."

[package.dependencies]
colorama = "0.4.4"

[metadata]
lock-version = "1.1"
python-versions = "^3.8"
content-hash = "7ae52da2736b4294be7a184e040cc78412add14e92b816077ede183f9e1c636c"

[metadata.files]
six = [\
    {file = "six-1.16.0-py2.py3-none-any.whl", hash = "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"},\
    {file = "six-1.16.0.tar.gz", hash = "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926"},\
]
colorama = [\
    {file = "colorama-0.4.4-py2.py3-none-any.whl", hash = "sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2"},\
    {file = "colorama-0.4.4.tar.gz", hash = "sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b"},\
]
```

#### Exclude devDependencies

IQ Server automatically excludes scanning devDependencies for projects using _poetry_ versions < 1.5.1

Due to removal of _category_ in the format of _poetry.lock_ file from version 1.5.1 onwards, older versions of IQ Server will not exclude the devDependencies while scanning projects using the new _poetry_ versions.

IQ Server will automatically exclude devDependencies for poetry versions 1.5.1 or higher, if pyproject.toml exists and is discoverable.

### Pipfile.lock

- From IQ CLI 2.0 onwards, `Pipfile.lock` files from Pipenv projects are also analyzed. This ensures direct and accurate identification of dependencies from Pipenv, eliminating the need for manual workarounds or CI configurations for Python projects.
- If you use the Jenkins plugin, note that you must upgrade the plugin version to take advantage of the new scanner capabilities introduced in IQ CLI 2.0.
- You can find [an example of a Pipfile.lock](https://github.com/pypa/pipfile?tab=readme-ov-file#pipfilelock) in the PyPA docs. Note that we only parse the dependencies under the `default` key, as those are the ones intended for production.

## Steps to analyze using the Sonatype IQ CLI

### Create requirements

Run pip freeze

pip freeze > requirements.txt

The requirements.txt encoding is UTF-8. Special note for Microsoft Windows users, the cmd.exe encoding may need to be changed to UTF-8. Please refer to Microsoft documentation on how to do this.

Example file content

| altgraph==0.10.2<br>backports-abc==0.5<br>backports.ssl-match-hostname==3.5.0.1<br>bdist-mpkg==0.5.0<br>certifi==2018.1.18<br>chardet==3.0.4<br>click==6.7<br>confire==0.2.0<br>Django==1.6<br>django-countries==3.3<br>django-make-app==0.1.3<br>docopt==0.6.2<br>enum34==1.1.6 |
| --- |

### Add environment markers (optional)

Adding [environment markers](https://peps.python.org/pep-0345/#environment-markers) can simplify the results by filtering out components that are not relevant to your deployment platform. Only the `sys_platform` environment marker is supported at the moment.

Add the environment marker next to the component(s) in the requirements.txt.

e.g.

| Django==1.6; sys_platform == 'win32' |
| --- |

### Run a scan

Invoke a Sonatype IQ CLI scan of the directory containing requirements.txt. Instructions on how to do this can be found here [Sonatype IQ CLI](https://help.sonatype.com/en/sonatype-iq-cli.html "Sonatype IQ CLI")

##   
Note: If your project uses Pipenv, you can instead rely on a `Pipfile.lock` being present. Starting with IQ CLI 2.0, the scanner automatically detects and analyzes dependencies from `Pipfile.lock`.