Judah’s way of developing for wordpress is definitely not wrong – simply different from the way I do things. Also, the two links he provided are very good. The blogpost appears worth a read and the codex is every wordpress developers bible.
To offer you some more options to look into, this is my take on wp development:
IDE / coding environment: If you wanted to alter say one element’s style, wp-admin (“the dashboard”) will suffice. For serious theme or plugin development however, it is definitely not an option.
Hence you should stick to your IDE of choice. For the sake of completeness, being a *nix user, I’d use Komodo Edit on Linux and Coda on Mac. However, whether your handling wordpress, another CMS or are coding from scratch is irrelevant to your choice of IDE – use what you feel comfortable with. One of the most basic requirements I have for an IDE – and I’d recommend you to look out for as well – is it’s capability to load and save files to a server via ftp directly. It gets tedious to save your files locally and upload them with a second app. Another important criterion is decent syntax highlighting for php, css, js and html.
Where to develop: After a bit of playing around you might come to the conclusion that you feel more comfortable developing locally – I don’t. I like to develop either directly on the server I will later go live on or on a server with the same setup (Apache and php versions, mySQL version to a lesser extent, last but not least: OS). Hence the importance of an IDE with integrated ftp capabilities. Should you choose to develop locally, make sure the environment is at least similar to the one you will later deploy the site on. For my way of doing things, I keep development subdomains on my server that are not accessible to the public. That way know the site will behave exactly the same way when I move it to go live.
WordPress theme development: Starting completely from scratch, i.e. with a blank theme folder and creating every required file yourself, would be a very time consuming and for the wordpress novice near impossible mission. As a basis for writing your first wordpress theme I’d highly recommend downloading the Starkers theme developed by the iconic Elliot Jay Stocks. Starkers contains almost no styling and no semantic markup. It however comes with all the essential functions and files of a standard wordpress theme. Today I use my own bare-bone theme template, basically Starkers with a number of modifications that I’d otherwise do to it for every theme. That template serves me well because it is adjusted to my personal way of doing things (and stripped of comments), but what Elliot did is provide the perfect starting point for the (aspiring) theme designer.
What else to do (if you go my path): Familiarize yourself with the basic structure of a wordpress theme. I wouldn’t go without a static header.php and footer.php that frame the site and inbetween of which all the magic happens. Know what best belongs in functions.php and why you should have one. Know what the wordpress loop is. Read the codex articles on Theme Development and The Loop. Read them again. When you understand a minimum 75% of their contents start dissecting Starkers, grasp its structure. Develop from there. Whenever you encounter a problem, check the codex (I do that mostly second hand – by googling the key terms followed by “wordpress”, often a page from the codex will be part of the first 5 results) first, then ask around.
One last resource: Two years ago, I would have highly recommended watching Chris Coyier’s (of css-tricks.com) three-part instructional video on wordpress theming. Given that it is from early 2009, it utilizes wp 2.7 (the current version being 3.2.1). I’d still recommend checking it out, it’s a good an easy-to-follow screencast series with a lot of advice that still applies. Just watch it keeping in mind that some things have progressed since then. The first video of the series can be found here, the others by googling, searching the site or altering the URL, respectively.