Add option to include docstrings with stubgen

[chylek]

Description

Closes #11965.

Add a --include-docstrings flag to stubgen. This was suggested in #11965 along with a use case.
When using this flag, the .pyi files will include docstrings for Python classes and functions and for C extension functions.
The flag is optional and does not change the default stubgen behaviour. When using the flag, the resulting function stubs that contain docstring will no longer be one-liners, but functions without a docstring still retain the default one-liner style.

Example input:

class A:
    """class docstring"""
    def func():
        """func docstring"""
        ...
    def nodoc():
        ...

output:

class A:
    """class docstring"""
    def func() -> None:
        """func docstring"""
        ...
    def nodoc() -> None: ...

Test Plan

Tests testIncludeDocstrings and testIgnoreDocstrings were added to test-data/unit/stubgen.test to ensure the code works as intended. All other tests passed as well.

C extension docstrings are tested using an updated bash script misc/test_stubgenc.sh with test data in test-data/pybind11_mypy_demo/stubgen-include-docs in same fashion as in an already existing test.

[sobolevn]

Thank you!

[chylek]

I gave it some more thought and I will have to fix another mistake - fastparse now keeps the docstrings in memory even when nothing uses them. I think I should also add include_docstrings internal flag to mypy.options. The docstrings would be retained only when stubgen (or anything that wants it) sets the flag.

Thanks @sobolevn for you comments, I'll fix it.
I was wondering about the C extension tests as well. There is a shell script that tests stubgenc as part of the github worklow. Perhaps I should create its duplicate that would test the docstrings.

[cnx-tcsikos]

@hauntsaninja Can you review this PR? I would definitely welcome this feature :)

[ws1088]

can you also consider adding variable docstring?

a_variable: int = 0
"""this is the docstring for a_variable"""

class Fields_Obj:
    DefaultValue=None
    """Get/set the default value of the data field"""

https://stackoverflow.com/questions/6060813/how-to-document-fields-and-properties-in-python

[chylek]

@ws1088 Interesting - attribute docstrings as PEP 224 were rejected but PEP 257 kind of brings them back. I think the AST doesn't make attribute docstrings accessible, so that wouldn't be an easy change. I think I won't be able to include attribute docstrings.

[FernandoLRubio]

Is there any ETA for this feature to be merged?

[ilevkivskyi]

@chylek Could you please resolve merge conflicts? This adds a frequently requested feature, so I would merge this (unless there are objections from other maintainers).

[ilevkivskyi]

Isn't this part completely repeats the other branch? If yes, then please use

if foo:
   bar
baz

instead of

if foo:
    bar
    baz
else:
    baz

[ilevkivskyi]

To be more precise, in this case you need:

bar
if foo:
    baz

since you always need a type, and docstring should come after (only if the flag is enabled)

[ilevkivskyi]

@chylek While a docs issue unrelated to your PR is being fixed, could you please fix a style issue I found?

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

[chylek]

Thank you for merging my pull request and for the guidance and feedback.