Web scraping redoc web api

Redoc is a React app which means the actual HTML is being built in runtime:

  • first the skeleton of the page loads, which also loads redoc javascript
  • then the Redoc downloads the OpenAPI json (or yaml) file and renders the actual HTML dynamically based on it

This is similar for many apps build with modern JS frameworks (vuejs, react, angular). To scrape these you have to actually load the page in a browser to run all the javascript.

I believe the most common way to do it nowadays is to use puppeteer (there is a python binding: https://github.com/pyppeteer/pyppeteer/)

Related Posts:

  1. How to find elements by class
  2. What is the meaning of [:] in python [duplicate]
  3. Understand the Find() function in Beautiful Soup
  4. TypeError: Object of type ‘bytes’ is not JSON serializable
  5. Difference between BeautifulSoup and Scrapy crawler?
  6. What are the most common Python docstring formats?
  7. Extract part of a regex match
  8. Python: What is the correct MIME type for attaching html file to email
  9. scrapy run spider from script
  10. H14 error in heroku – “no web processes running”
  11. How to save an image locally using Python whose URL address I already know?
  12. can we use XPath with BeautifulSoup?
  13. What should I use to open a url instead of urlopen in urllib3
  14. Can I remove script tags with BeautifulSoup?
  15. Python code to remove HTML tags from a string 
  16. How to read html from a url in python 3
  17. How do I lowercase a string in Python?
  18. Difference between del, remove, and pop on lists
  19. How to use the pass statement
  20. How do you round UP a number?
  21. How do I update\upgrade pip itself from inside my virtual environment?
  22. ImportError: DLL load failed: The specified module could not be found
  23. Replacements for switch statement in Python?
  24. Correct way to write line to file?
  25. How to read a file line-by-line into a list?
  26. How do I sort a dictionary by value?
  27. TypeError: ‘NoneType’ object is not iterable in Python
  28. Relative imports – ModuleNotFoundError: No module named x
  29. ValueError: invalid literal for int() with base 10: ”
  30. What does the “yield” keyword do?
  31. How do I get the number of elements in a list?
  32. What does hash do in python?
  33. How to copy a dictionary and only edit the copy
  34. Creating a static class with no instances
  35. How to avoid “RuntimeError: dictionary changed size during iteration” error?
  36. if/else in a list comprehension
  37. Could not find a version that satisfies the requirement tensorflow
  38. Creating an empty Pandas DataFrame, then filling it?
  39. Finding the average of a list
  40. How to solve dictionary changed size during iteration error?
  41. iloc giving ‘IndexError: single positional indexer is out-of-bounds’
  42. What does “\r” do in the following script?
  43. Anaconda vs. miniconda
  44. No module named serial
  45. ImportError: No module named psycopg2
  46. DataFrame constructor not properly called
  47. How does functools partial do what it does?
  48. What is the problem with shadowing names defined in outer scopes?
  49. TypeError: write() argument must be str, not bytes (Python 3 vs Python 2 )
  50. How to clear the interpreter console?
  51. How to make a class JSON serializable
  52. What does the ‘b’ character do in front of a string literal?
  53. How can I upgrade NumPy?
  54. How to get element-wise matrix multiplication (Hadamard product) in numpy?
  55. Matplotlib automatic legend outside plot
  56. Why am I getting this syntax error: keyword can’t be an expression
  57. Django: ImproperlyConfigured: The SECRET_KEY setting must not be empty
  58. What is dtype(‘O’), in pandas?
  59. ctypes – Beginner
  60. Keras, how do I predict after I trained a model?
  61. How do I find out my PYTHONPATH using Python?
  62. ModuleNotFoundError: No module named ‘MySQLdb’
  63. How to change dataframe column names in pyspark?
  64. Normalize data in pandas
  65. Can’t broadcast input array from shape (3,1) into shape (3,)
  66. How to create an array of bits in Python?
  67. Setting Django up to use MySQL
  68. In Python, when to use a Dictionary, List or Set?
  69. How can I check for NaN values?
  70. Python – how to use the constant e^
  71. Importing variables from another file in Python
  72. How to read a file without newlines?
  73. Encrypt & Decrypt using PyCrypto AES 256
  74. Is there a list of Pytz Timezones?
  75. Flask example with POST
  76. How do I change the figure size with subplots?
  77. can’t open tensorboard 0.0.0.0:6006 or localhost:6006
  78. ImportError: No module named six
  79. Easy pretty printing of floats?
  80. Algorithm for solving Sudoku
  81. How to read pickle file?
  82. How does str(list) work?
  83. Automatic indentation for Python in Notepad++
  84. pandas groupby sort within groups
  85. Python Socket Receive Large Amount of Data
  86. How to find out whether a file is at its `eof`?
  87. TypeError: Image data can not convert to float
  88. coercing to Unicode: need string or buffer, NoneType found when rendering in django admin
  89. Convert from ASCII string encoded in Hex to plain ASCII?
  90. Simple way to create matrix of random numbers
  91. pandas: multiple conditions while indexing data frame – unexpected behavior
  92. How can I tail a log file in Python?
  93. How to write unicode strings into a file?
  94. python-How to set global variables in Flask?
  95. How do I install pip on macOS or OS X?
  96. NameError: name ‘get_ipython’ is not defined
  97. Equivalent to matlab’s imagesc in matplotlib? [duplicate]
  98. bash: syntax error near unexpected token `(‘ – Python
  99. Find all files in a directory with extension .txt in Python
  100. “public” or “private” attribute in Python ? What is the best way?

Leave a Comment