Developing Your Own Scientific Python Code
In many cases, scientific research takes you into totally new areas of knowledge, never before explored by others. This means the computational work you need to do may be totally new as well. Although typically such code development still happens in C or FORTRAN, Python is growing in popularity. This is especially true in physics.
If you are just working on a small code base, a basic text editor is fine; however, once your project reaches a certain size, using a proper IDE is a huge advantage. Luckily, an open-source project seeks to fill this exact niche: Spyder. Spyder is available for Windows, Mac OS X and Linux. You should be able to find a package for your distribution, but if not, you always can grab the binaries or source code at the main Spyder Web site.
Spyder actually is written in Python, and it has been designed with a plugin architecture. This means you can add extra functionality by installing plugins. If you can't find a plugin for the functionality you need, you always can write your own.
When you start Spyder, several panes open and a temporary Python script that you can use to start editing appears. The main pane is the editor, where the temporary script is loaded and ready for you to start working. The right-hand side of the window is broken into two more panes. The bottom pane is the console to a running Python interpreter. Here you can see that Spyder automatically loads NumPy, SciPy and matplotlib on startup, so you already have most of the tools you likely will need ready to go.
Figure 1. When you start Spyder, several information panes along with the main editor pane appear.
You can use this console just as you would any other Python interpreter. The top pane has multiple tabs. The first tab that opens at start up is an object inspector. This tab lets you look at the details of any objects being used in your code. The other tabs available are a variable inspector and a file explorer.
Figure 2. The object inspector lets you look at the details of any objects you may want to use.
Let's begin by looking at the main editor pane. Like any other programming editor, Spyder provides full-color highlighting of Python syntax. Rope is used to provide code introspection capabilities to the editor. If you start typing a function call, Spyder makes suggestions for code completion.
Pyflakes provides on-the-fly code analysis. Any errors in your code are highlighted right away with a triangle symbol in the margin. When you hover over it, details for the error are displayed in a pop-up window. You also can set breakpoints in the editor that are used by the Python debugger. This way, you have a bit of control over how your code runs when you run it under the pdb debugger.
Figure 3. You can set all of the options for the editor window in the preferences section.
The console pane provides a full set of tools for controlling multiple interpreters. When you run a script from the editor, you have the option of starting up a new interpreter in which to run it. Or, you can run it in an existing interpreter. You can set this behavior in Spyder's preferences. You also simply can create a new interpreter from the console pane directly.
Figure 4. The preferences for the console section control things like whether it is monitored and so forth.
Also interesting is that you can create IPython interpreters in the console. Then, you have all of the extra functionality provided by IPython at your disposal. One of these advanced features is the ability to use multiple IPython engines in parallel. From the console in Spyder, you can create these IPython engines that will run in the background, ready to be used for whatever parallel processing you may have in mind. So, not only can you develop your new scientific code as a parallel program, you also can work with it directly from Spyder, in parallel. All of these extra interpreters and engines run as separate processes, which means they will not affect Spyder itself or cause it to hang if something bad happens within one of the Python interpreters.
One of the extra tabs in the top-right pane brings up the variable explorer. This pane gives you a list of all the variables currently active in the memory space within Spyder. It shows the name, type, size and value for each of the variables accessible in the global namespace. This applies to both the internal Python interpreter and any external ones. The variable explorer can handle all of the standard data types, like strings, integers and floats. It also includes an array editor that can be used to edit lists and tuples. The array editor provides a nice environment for editing complex data types.
Figure 5. The variable explorer gives you a list of everything within the global namespace.
Spyder provides even more tools though. By right-clicking on a list or tuple, you can do some basic data analysis. Spyder allows you to plot the values in the data object to see what it looks like. Or, you can look at a histogram of the values if a statistical analysis would make more sense.
Figure 6. Spyder lets you plot the data within a list or tuple.
All of these tools are handy, but by themselves, they aren't enough if you are developing a large code base. In such cases, you will need some kind of project-level organization. Spyder can help in that situation too. You can create a project in order to encapsulate a group of files as a single unit. Creating a new project makes a new folder to store all of the associated files. Opening this new project creates a new pane where you can work with the project files. By right-clicking on the project, you can create a new file, folder, module or package. When you create a new file, Spyder opens it up in the editor, ready for you to start working on it.
Figure 7. If you are working on a larger code base, the project explorer will come in very handy.
The last tool you should be aware of is the profiler. The first step in program development is writing code that works. After that, your job is to write code that is as efficient as possible. The rule of thumb is to optimize last, and only what needs to be optimized. But, what part of your program is that? Without reliable measurements, you won't know what needs optimizing. In Spyder, you can click on the menu item Run→Profile or press F10. This runs your code under the Python profiler, giving you a breakdown of where all the time is being spent. Once you have this information, you can focus your energies where they will do the most good.
Figure 8. The profiler opens a new pane at the bottom of the window, showing you numbers of calls and time for each function.
Hopefully, you can take Spyder and run with it while developing your own scientific Python code. Although lots of IDEs exist for developing code, there aren't very many setups geared toward developing scientific code. With Spyder, you should have a head start in developing your new breakthrough code, solving the problem that could win you the next Nobel Prize.
Joey Bernard has a background in both physics and computer science. This serves him well in his day job as a computational research consultant at the University of New Brunswick. He also teaches computational physics and parallel programming.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- My +1 Sword of Productivity
- Non-Linux FOSS: Caffeine!
- Managing Linux Using Puppet
- Doing for User Space What We Did for Kernel Space
- SuperTuxKart 0.9.2 Released
- Parsing an RSS News Feed with a Bash Script
- Google's SwiftShader Released
- Rogue Wave Software's Zend Server
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide